基于循环图注意力强化学习的交叉口多车协同控制方法

doi:10.3778/j.issn.1002-8331.2406-0105

摘要/Abstract

摘要： 在未来一段时间内，联网自动驾驶汽车（connected automated vehicle，CAV）和人工驾驶车辆（human-drive vehicle，HV）将会在无信号交叉口混合行驶。由于CAV观测范围存在限制，仅能根据其邻域内的混合交通流信息做出决策，将无信号交叉口的多车协同控制过程建模为分布式可观测马尔可夫决策过程，并建立基于柔性演员-评论家（soft actor-critic，SAC）的集中式训练分布式执行（centralized training and decentralized execution，CTDE）框架。将车辆之间的关系建模成图，以多层注意力层作为演员网络和评论家网络的卷积核，推断CAV邻域内车辆的图特征。构建门控循环单元（gated recurrent unit，GRU）加强邻域动态图特征的长期记忆，避免车辆移动过程中邻域信息高度变化导致的信息遗忘。仿真结果表明，在无碰撞的前提下，较现有最优的无信号交叉口分布式控制算法，所提算法在单向交叉口、可转向交叉口、2×2交叉口网络场景下的平均车速分别提升了10.51%，4.64%，10.24%。

关键词: 无信号交叉口, 车辆控制, 多智能体强化学习, 自动驾驶, 局部可观测

Abstract: There will be mixed traffic consisting of connected automated vehicles (CAVs) and human-drive vehicles (HVs) at future unsignalized intersections for long periods of time. CAV can only make decisions based on neighborhood information of mixed traffic due to CAV’s partial observation. The process of multi-vehicle cooperative control at unsignalized intersections is modeled as a decentralized partially observable Markov decision process. This study proposes a centralized training and decentralized execution (CTDE) framework based on soft actor-critic (SAC). The relationship between vehicles is modeled into a graph, and multiple attention layers are used as the convolution kernel of actor network and critic network to infer the neighbor graph features of vehicles. Gated recurrent unit (GRU) is developed to keep the long-term memory of neighbor dynamic graph features and avoid information for-getting caused by the changes of neighborhood information during vehicle movement. Simulation results show that the collision rate of this algorithm is 0. The average vehicle speed of one-way 1×1 intersection, multi-way 1×1 intersection, and multi-way 2×2 intersection network is increased by 10.51%, 4.64%, and 10.24% compared with the state-of-the-art multi-vehicles cooperative control algorithms, respectively.

Key words: unsignalized intersection, vehicle control, multi-agent reinforcement learning, autonomous driving, partially observable

杨伟达, 吴志周, 梁韵逸. 基于循环图注意力强化学习的交叉口多车协同控制方法[J]. 计算机工程与应用, 2025, 61(19): 282-291.

YANG Weida, WU Zhizhou, LIANG Yunyi. Multi-Vehicle Cooperative Control at Intersection：Recurrent Graph Attention Reinforcement Learning[J]. Computer Engineering and Applications, 2025, 61(19): 282-291.

参考文献

[1] MOUSAVI S M, OSMAN O A, LORD D, et al. Investigating the safety and operational benefits of mixed traffic environments with different automated vehicle market penetration rates in the proximity of a driveway on an urban arterial[J]. Accident Analysis & Prevention, 2021, 152: 105982.
[2] CHEN S Z, HU X H, ZHAO J H, et al. A review of decision-making and planning for autonomous vehicles in intersection environments[J]. World Electric Vehicle Journal, 2024, 15(3): 99.
[3] ISELE D, RAHIMI R, COSGUN A, et al. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation. Piscataway: IEEE, 2018: 2034-2039.
[4] LU Q, KIM K D. A mixed integer programming approach for autonomous and connected intersection crossing traffic control[C]//Proceedings of the 2018 IEEE 88th Vehicular Technology Conference. Piscataway: IEEE, 2018: 1-6.
[5] DRESNER K, STONE P. A multiagent approach to autonomous intersection management[J]. Journal of Artificial Intelligence Research, 2008, 31: 591-656.
[6] 季亚平. 面向自动驾驶车辆队列的无信号交叉口协同控制策略研究[D]. 南京: 东南大学, 2022.
JI Y P. Research on cooperative control strategies of the non-signalized crossing for autonomous vehicle platoons[D]. Nanjing: Southeast University, 2022.
[7] FATHOLLAHNEJAD N, VILLANI E, PATHAN R, et al. On reliability analysis of leader election protocols for virtual traffic lights[C]//Proceedings of the 2013 43rd Annual IEEE/IFIP Conference on Dependable Systems and Networks Workshop. Piscataway: IEEE, 2013: 1-12.
[8] 胡梦岩. 智能网联环境下的无信号交叉口冲突消解方法研究[D]. 西安: 长安大学, 2017.
HU M Y. Centralized conflict resolution at unsignalized intersections based on connected and automated vehicle[D]. Xi’an: Chang’an University, 2017.
[9] YU C H, SUN W L, LIU H X, et al. Managing connected and automated vehicles at isolated intersections: from reservation- to optimization-based methods[J]. Transportation Research Part B: Methodological, 2019, 122: 416-435.
[10] MA M T, LI Z X. A speed-maximization trajectory optimization model on a reservation-based intersection control system[J]. Transportation Research Part C: Emerging Technologies, 2023, 154: 104266.
[11] 王润民, 凡海金, 何佳浚, 等. 无信控交叉口网联车辆动态碰撞风险检测与预警策略[J]. 计算机工程与应用, 2024, 60(13): 330-337.
WANG R M, FAN H J, HE J J, et al. Dynamic collision risk detection and early warning strategy of connected vehicles at unsignalized intersections[J]. Computer Engineering and Applications, 2024, 60(13): 330-337.
[12] LI G, WU J, HE Y. HARL: a novel hierachical adversary reinforcement learning for autonomous intersection management[J]. arXiv:2205.02428, 2022.
[13] GUO Z H, WU Y, WANG L F, et al. Coordination for connected and automated vehicles at non-signalized intersections: a value decomposition-based multiagent deep reinforcement learning approach[J]. IEEE Transactions on Vehicular Technology, 2023, 72(3): 3025-3034.
[14] LI D W, ZHU F, CHEN T Y, et al. COOR-PLT: a hierarchical control model for coordinating adaptive platoons of connected and autonomous vehicles at signal-free intersections based on deep reinforcement learning[J]. Transportation Research Part C: Emerging Technologies, 2023, 146: 103933.
[15] GUNARATHNA U, KARUNASEKERA S, BOROVICA-GAJIC R, et al. Real-time intelligent autonomous intersection management using reinforcement learning[C]//Proceedings of the 2022 IEEE Intelligent Vehicles Symposium. Piscataway: IEEE, 2022: 135-144.
[16] CHEN J, SUGUMARAN V, QU P Y. Connected and automated vehicle control at unsignalized intersection based on deep reinforcement learning in vehicle-to-infrastructure environment[J]. International Journal of Distributed Sensor Networks, 2022, 18(7): 155013292211140.
[17] ANTONIO G P, MARIA-DOLORES C. Multi-agent deep reinforcement learning to manage connected autonomous vehicles at tomorrow’s intersections[J]. IEEE Transactions on Vehicular Technology, 2022, 71(7): 7033-7043.
[18] YAN Z X, WU C. Reinforcement learning for mixed autonomy intersections[C]//Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference. Piscataway: IEEE, 2021: 2089-2094.
[19] WU C, KREIDIEH A, PARVATE K, et al. Flow: architecture and benchmarking for reinforcement learning in traffic control[J]. arXiv:1710.05465, 2017.
[20] PENG B L, KESKIN M F, KULCSáR B, et al. Connected autonomous vehicles for improving mixed traffic efficiency in unsignalized intersections with deep reinforcement learning[J]. Communications in Transportation Research, 2021, 1: 100017.
[21] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[J]. arXiv:1409.0473, 2014.
[22] CHO K, VAN MERRI?NBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv:1406.1078, 2014.
[23] HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of the International Conference on Machine Learning, 2018: 1861-1870.
[24] QUANG TRAN D, BAE S H. Proximal policy optimization through a deep reinforcement learning framework for multiple autonomous vehicles at a non-signalized intersection[J]. Applied Sciences, 2020, 10(16): 5722.
[25] YU C, VELU A, VINITSKY E, et al. The surprising effectiveness of MAPPO in cooperative, multi-agent games[J]. arXiv:2103.01955, 2021.
[26] ACKERMANN J, GABLER V, OSA T, et al. Reducing overestimation bias in multi-agent domains using double centralized critics[J]. arXiv:1910.01465, 2019.
[27] IQBAL S, SHA F. Actor-attention-critic for multi-agent reinforcement learning[C]//Proceedings of the 36th International Conference on Machine Learning, 2019: 2961-2970.
[28] JIANG J, DUN C, HUANG T, et al. Graph convolutional reinforcement learning[J]. arXiv:1810.09202, 2018.
[29] LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. arXiv:1509.02971, 2015.
[30] LOWE R, WU Y I, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Advances in Neural Information Processing Systems, 2017.