动态环境下共融机器人深度强化学习导航算法

doi:10.3778/j.issn.1002-8331.2405-0088

摘要/Abstract

摘要： 在过去的几十年里，移动服务机器人的导航算法得到了广泛研究，但智能体仍然缺乏人类在拥挤环境中展现出的复杂性和合作性。随着人机共融的应用不断拓展，机器人和人类共享工作空间的协作将愈发重要，因此下一代移动服务机器人需要符合社交要求，才能被人类接受。为了提升多智能体在动态场景中的自主导航能力，针对多智能体导航中社会适应性低和寻找最优值函数问题，提出了一种动态环境下共融机器人深度强化学习避障算法。建立了更贴近人类行为的运动模型并将其添加到深度强化学习框架中，用于提高共融机器人的合作性；为了在行人物理安全的基础上提升其感知安全，重新制定了奖励函数；利用非线性深度神经网络代替传统的值函数，解决寻找最优值函数问题。仿真实验显示，相较于最新的深度强化学习导航方法，该方法在不增加导航时间的情况下实现了100%的导航成功率，且没有发生任何碰撞。结果表明，该方法使共融机器人最大限度地满足人类的社交原则，同时朝着目标前进，有效提高了行人的感知安全。

关键词: 服务机器人, 避障算法, 深度强化学习, 最优值函数, 奖励函数

Abstract: In the past few decades, navigation algorithms for mobile service robots have been extensively studied, but intelligent agents still lack the complexity and cooperation exhibited by humans in crowded environments. With the continuous expansion of human-machine integration applications, collaboration between robots and humans in shared workspaces will become increasingly important. Therefore, the next generation of mobile service robots needs to meet social requirements in order to be accepted by humans. In order to enhance the autonomous navigation ability of multi-agent systems in dynamic scenarios, a deep reinforcement learning obstacle avoidance algorithm for coexisting-cooperative-cognitive robots in dynamic environments is proposed to address the issues of low social adaptability and finding the optimal value function in multi-agent navigation. A motion model that is more closely related to human behavior is established and added to the deep reinforcement learning framework to improve the cooperation of coexisting-cooperative-cognitive robots. In order to enhance the perceived safety of pedestrians based on their physical safety, a reward function is redefined. Nonlinear deep neural networks are used instead of traditional value functions to solve the problem of finding the optimal value function. Simulation experiments show that compared to the latest deep reinforcement learning navigation methods, the proposed method achieves a 100% navigation success rate without increasing navigation time and without any collisions. The results indicate that this method maximizes the satisfaction of human social principles for the fusion robot, while effectively moving towards the goal and improving the perceived safety of pedestrians.

Key words: service robot, obstacle avoidance, deep reinforcement learning, optimal value function, reward function

顾金浩, 况立群, 韩慧妍, 曹亚明, 焦世超. 动态环境下共融机器人深度强化学习导航算法[J]. 计算机工程与应用, 2025, 61(4): 90-98.

GU Jinhao, KUANG Liqun, HAN Huiyan, CAO Yaming, JIAO Shichao. Deep Reinforcement Learning Navigation Algorithm for Coexisting-Cooperative-Cognitive Robots in Dynamic Environment[J]. Computer Engineering and Applications, 2025, 61(4): 90-98.

参考文献

[1] HUANG D, CHEN Q, HUANG S, et al. Consumer intention to use service robots: a cognitive-affective-conative framework[J]. International Journal of Contemporary Hospitality Management, 2024, 36(6): 1893-1913.
[2] LI B T. Tri-Co robots for the future: an interview with Guang-Zhong Yang[J]. National Science Review, 2023, 10(5): nwac263.
[3] DING H. Preface to special topic on Tri-Co robots[J]. National Science Review, 2023, 10(5): nwad037.
[4] 王志军, 刘璐, 李占贤. 共融机器人综述及展望[J]. 制造技术与机床, 2020(6): 30-38.
WANG Z J, LIU L, LI Z X. A review and outlook of coexisting-cooperative-cognitive robots[J]. Manufacturing Technology & Machine Tool, 2020(6): 30-38.
[5] AKALIN N, KRISTOFFERSSON A, LOUTFI A. Do you feel safe with your robot? Factors influencing perceived safety in human-robot interaction based on subjective and objective measures[J]. International Journal of Human-Computer Studies, 2022, 158: 102744.
[6] VAN DEN BERG J, GUY S J, LIN M, et al. Reciprocal n-body collision avoidance[C]//Proceedings of the 14th International Symposium on Robotics Research. Berlin, Heidelberg: Springer, 2011: 3-19.
[7] LIU S, CHANG P, HUANG Z, et al. Intention aware robot crowd navigation with attention-based interaction graph[C]//Proceedings of the 2023 IEEE International Conference on Robotics and Automation. Piscataway: IEEE, 2023: 12015-12021.
[8] LI Y, HU X, ZHUANG Y, et al. Deep reinforcement learning (DRL): another perspective for unsupervised wireless localization[J]. IEEE Internet of Things Journal, 2019, 7(7): 6279-6287.
[9] MARTINEZ-BASELGA D, RIAZUELO L, MONTANO L. Improving robot navigation in crowded environments using intrinsic rewards[C]//Proceedings of the 2023 IEEE International Conference on Robotics and Automation. Piscataway: IEEE, 2023: 9428-9434.
[10] SAMSANI S S, MUTAHIRA H, MUHAMMAD M S. Memory-based crowd-aware robot navigation using deep reinforcement learning[J]. Complex & Intelligent Systems, 2023, 9(2): 2147-2158.
[11] CHEN C, LIU Y, KREISS S, et al. Crowd-robot interaction: crowd-aware robot navigation with attention-based deep reinforcement learning[C]//Proceedings of the 2019 International Conference on Robotics and Automation, 2019: 6015-6022.
[12] SHI W, ZHOU Y, ZENG X, et al. Enhanced spatial attention graph for motion planning in crowded, partially observable environments[C]//Proceedings of the 2022 International Conference on Robotics and Automation, 2022: 4750-4756.
[13] VAN DER HEIDEN T, MIRUS F, VAN HOOF H. Social navigation with human empowerment driven deep reinforcement learning[C]//Proceedings of the 29th International Conference on Artificial Neural Networks. Cham: Springer, 2020: 395-407.
[14] VOLPI N C, POLANI D. Goal-directed empowerment: combining intrinsic motivation and task-oriented behavior[J]. IEEE Transactions on Cognitive and Developmental Systems, 2020, 15(2): 361-372.
[15] CHEN C, HU S, NIKDEL P, et al. Relational graph learning for crowd navigation[C]//Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2020: 10007-10013.
[16] ZHOU Z, ZHU P, ZENG Z, et al. Robot navigation in a crowd by integrating deep reinforcement learning and online planning[J]. Applied Intelligence, 2022, 52(13): 15600-15616.
[17] VEMULA A, MUELLING K, OH J. Social attention: modeling attention in human crowds[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation. Piscataway: IEEE, 2018: 4601-4607.
[18] ZHOU X, PIAO S, CHI W, et al. HeR-DRL: heterogeneous relational deep reinforcement learning for decentralized multi-robot crowd navigation[J]. arXiv:2403.10083, 2024.
[19] SIGAL A, LIN H C, MOON A J. Improving reinforcement learning training regimes for social robot navigation[J]. arXiv:2308.14947, 2023.
[20] TUAN P M, TAI N D, HUY T Q, et al. Flexible path planning of mobile robot for avoiding the dynamic obstacles using fuzzy controllers[J]. International Journal of Mechanical Engineering and Robotics Research, 2024, 13(1): 126-132.
[21] ESCUDIE E, MATIGNON L, SARAYDARYAN J. Attention graph for multi-robot social navigation with deep reinforcement learning[J]. arXiv:2401.17914, 2024.
[22] HELBING D, MOLNAR P. Social force model for pedestrian dynamics[J]. Physical review E, 1995, 51(5): 4282.
[23] CHEN Y F, LIU M, EVERETT M, et al. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning[C]//Proceedings of the 2017 IEEE International Conference on Robotics and Automation. Piscataway: IEEE, 2017: 285-292.
[24] FAN T, LONG P, LIU W, et al. Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios[J]. The International Journal of Robotics Research, 2020, 39(7): 856-892.
[25] EVERETT M, CHEN Y F, HOW J P. Collision avoidance in pedestrian-rich environments with deep reinforcement learning[J]. IEEE Access, 2021, 9: 10357-10377.