改进MADDPG算法的未知环境下多智能体单目标协同探索

doi:10.3778/j.issn.1002-8331.2408-0131

摘要/Abstract

摘要： 针对多智能体深度确定性策略梯度算法（multi-agent deep deterministic policy gradient，MADDPG）在未知环境下探索效率低下的问题，提出多智能体深度强化学习算法RE-MADDPG-C。利用残差网络（residual network，ResNet）缓解网络中的梯度消失和梯度爆炸问题，提高算法的收敛速度。为解决未知环境下单目标探索中奖励稀疏导致的收敛困难问题，引入多智能体内在好奇心模块（intrinsic curiosity module，ICM），将好奇心奖励作为智能体的内在奖励，为其提供额外的探索动机。通过设计合理的探索奖励函数，使得多智能体能够在未知环境下完成单目标探索任务。仿真实验结果表明，该算法在训练阶段获得的奖励提升更快，能够快速完成探索任务，相比MADDPG及其他算法训练时间缩短，且获得的全局平均奖励更高。

关键词: 深度强化学习, RE-MADDPG-C, 残差网络, 内在好奇心模块（ICM）, 奖励稀疏

Abstract: To address the inefficiency of the multi-agent deep deterministic policy gradient (MADDPG) algorithm in unknown environments, a new multi-agent deep reinforcement learning algorithm called RE-MADDPG-C is proposed. This algorithm uses residual networks (ResNet) to alleviate gradient vanishing and explosion issues, enhancing convergence speed. To tackle the convergence difficulty caused by sparse rewards in single-goal exploration in unknown environments, a multi-agent intrinsic curiosity module (ICM) is introduced. The curiosity reward serves as an intrinsic motivation for agents, providing additional exploration incentives. By designing a suitable exploration reward function, agents can accomplish single-goal tasks in unknown environments. Simulation results show that the proposed algorithm achieves faster reward improvement during training, quickly completing exploration tasks. Compared to MADDPG and other algorithms, the proposed algorithm reduces training time and achieves higher global average rewards.

Key words: deep reinforcement learning, RE-MADDPG-C, residual network, intrinsic curiosity module (ICM), sparse rewards

韩慧妍, 石树熙, 况立群, 韩燮, 熊风光. 改进MADDPG算法的未知环境下多智能体单目标协同探索[J]. 计算机工程与应用, 2025, 61(22): 320-328.

HAN Huiyan, SHI Shuxi, KUANG Liqun, HAN Xie, XIONG Fengguang. Multi-Agent Single-Goal Collaborative Exploration in Unknown Environments with Improving MADDPG Algorithm[J]. Computer Engineering and Applications, 2025, 61(22): 320-328.

参考文献

[1] 祝泽亚, 谢君, 王智. 未知环境下双机器人协同探索方法[J]. 电光与控制, 2020, 27(4): 32-37.
ZHU Z Y, XIE J, WANG Z. A method of dual robot collaborative exploration in unknown environment[J]. Electronics Optics & Control, 2020, 27(4): 32-37.
[2] 宁宇铭, 李团结, 姚聪, 等. 基于快速扩展随机树-贪婪边界搜索的多机器人协同空间探索方法[J]. 机器人, 2022, 44(6): 708-719.
NING Y M, LI T J, YAO C, et al. Multi-robot cooperative space exploration method based on rapidly-exploring random trees and greedy frontier-based exploration[J]. Robot, 2022, 44(6): 708-719.
[3] TANG J, LIU G, PAN Q T. A review on representative swarm intelligence algorithms for solving optimization problems: applications and trends[J]. IEEE/CAA Journal of Automatica Sinica, 2021, 8(10): 1627-1643.
[4] MIR I, GUL F, MIR S, et al. Multi-agent variational approach for robotics: a bio-inspired perspective[J]. Biomimetics, 2023, 8(3): 294.
[5] GARG V, SHUKLA A, TIWARI R. AERPSO: an adaptive exploration robotic PSO based cooperative algorithm for mult-iple target searching[J]. Expert Systems with Applications, 2022, 209: 118245.
[6] SUTTON R S, BARTO A G. Reinforcement learning: an introduction[M]. Cambridge: MIT Press, 1998.
[7] ZHOU Q, LIAN Y, WU J Y, et al. An optimized Q-Learning algorithm for mobile robot local path planning[J]. Knowledge-Based Systems, 2024, 286: 111400.
[8] LIU I J, JAIN U, YEH R A, et al. Cooperative exploration for multi-agent deep reinforcement learning[C]//Proceedings of the 38th International Conference on Machine Learning, 2021: 6826-6836.
[9] OROOJLOOY A, HAJINEZHAD D. A review of cooperative multi-agent deep reinforcement learning[J]. Applied Intelligence, 2023, 53(11): 13677-13722.
[10] LI T Y, SHI D X, WANG Z, et al. Learning cooperative strategies in multi-agent encirclement games with faster prey using prior knowledge[J]. Neural Computing and Applications, 2024, 36(25): 15829-15842.
[11] 马佩鑫, 程钰, 侯健, 等. 基于多智能体深度强化学习的协作导航应用[J]. 计算机系统应用, 2023, 32(8): 95-104.
MA P X, CHENG Y, HOU J, et al. Cooperative navigation application based on multi-agent deep reinforcement learning[J]. Computer Systems and Applications, 2023, 32(8): 95-104.
[12] WANG X, WANG S, LIANG X X, et al. Deep reinforcement learning: a survey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(4): 5064-5078.
[13] HAO J Y, YANG T P, TANG H Y, et al. Exploration in deep reinforcement learning: from single-agent to multiagent dom-ain[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(7): 8762-8782.
[14] LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Advances in Neural Information Processing Systems 30, 2017.
[15] LIU W Z, DONG L, NIU D, et al. Efficient exploration for multi-agent reinforcement learning via transferable successor features[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(9): 1673-1686.
[16] WAN K F, WU D W, LI B, et al. ME-MADDPG: an efficient learning-based motion planning method for multiple agents in complex environments[J]. International Journal of Intelligent Systems, 2022, 37(3): 2393-2427.
[17] LI B, WANG J M, SONG C, et al. Multi-UAV roundup strategy method based on deep reinforcement learning CEL-MADDPG algorithm[J]. Expert Systems with Applications, 2024, 245: 123018.
[18] XUE Y T, CHEN W S. Multi-agent deep reinforcement learning for UAVs navigation in unknown complex environment[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(1): 2290-2303.
[19] GU S D, KUBA J G, CHEN Y P, et al. Safe multi-agent reinforcement learning for multi-robot control[J]. Artificial Intelligence, 2023, 319: 103905.
[20] LI J, SHI X X, LI J H, et al. Random curiosity-driven exploration in deep reinforcement learning[J]. Neurocomputing, 2020, 418: 139-147.
[21] DAI T H, DU Y L, FANG M, et al. Diversity-augmented intrinsic motivation for deep reinforcement learning[J]. Neurocomputing, 2022, 468: 396-406.
[22] PATHAK D, AGRAWAL P, EFROS A A, et al. Curiosity-driven exploration by self-supervised prediction[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2017: 488-489.
[23] BURDA Y, EDWARDS H, STORKEY A, et al. Exploration by random network distillation[J]. arXiv:1810.2894, 2018.
[24] ZHENG L L, CHEN J R, WANG J H, et al. Episodic multi-agent reinforcement learning with curiosity-driven exploration[C]//Advances in Neural Information Processing Systems 34, 2021: 3757-3769.
[25] XU F C, KANEKO T. Curiosity-driven exploration for cooperative multi-agent reinforcement learning[C]//Proceedings of the 2023 International Joint Conference on Neural Networks. Piscataway: IEEE, 2023: 1-8.
[26] ALACAOGLU A, VIANO L, HE N, et al. A natural actor-critic framework for zero-sum Markov games[C]//Proceedings of the 39th International Conference on Machine Learning, 2022: 307-366.
[27] BORAWAR L, KAUR R. ResNet: solving vanishing gradient in deep networks[C]//Proceedings of International Conference on Recent Trends in Computing. Singapore: Springer, 2023: 235-247.
[28] YU C, VELU A, VINITSKY E, et al. The surprising effectiveness of PPO in cooperative, multi-agent games[C]//Advances in Neural Information Processing Systems 35, 2022: 24611-24624.
[29] XING X J, ZHOU Z W, LI Y, et al. Multi-UAV adaptive cooperative formation trajectory planning based on an improved MATD3 algorithm of deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2024, 73(9): 12484-12499.
[30] LI J, GAJANE P. Curiosity-driven exploration in sparse-reward multi-agent reinforcement learning[J]. arXiv:2302. 10825, 2023.