Multi-Agent Reinforcement Learning for On-Ramp Merging Control

doi:10.3778/j.issn.1002-8331.2308-0435

Abstract

Abstract: On-ramp merging is a challenging task for CAV, considering that mixed traffic with CAV and manually driven vehicles HDVs will exist in more traffic scenarios in the coming time. Based on the traffic characteristics of the merging area, the multi-vehicle cooperative convergence is represented as a Markov decision process, and a reward function that considers both vehicle safety and efficiency is established. The CTDE-MARL algorithm framework is proposed based on the distributed MARL framework, which saves computational resources on a single intelligence. Two control algorithms, A2C and PPO, are established based on the two frameworks. Finally, the results of simulation experiments show that the overall performance of the established improved algorithm is better than the original algorithm, which improves the average speed of vehicles and satisfies the minimum headway while reducing the collision rate and the waiting time for merging. It satisfies the safety and improves the efficiency of vehicles in the merging area.

Key words: reinforcement learning, merging area, automatic driving, multi-agents

摘要： 对于智能网联车辆（connected automated vehicle，CAV）来说，上匝道合并是一项具备挑战性的任务，考虑到未来一段时间内有CAV和人工驾驶车辆（human-drive vehicle，HAV）的混合交通将存在于更多的交通场景中。根据合流区交通特性，将多车协同汇入表示为马尔可夫决策过程，建立了同时考虑车辆安全和效率的奖励函数。基于分布式多智能体强化学习（muti-agent reinforcement learning，MARL）框架提出中心式训练分散式执行的改进框架（centralized training and decentralized execution，CTDE）的MARL算法框架，节省了单智能体上的计算资源。建立基于两种框架的优势动作评论家（advantage actor critic，A2C）和近端策略优化（proximal policy optimization，PPO）两种控制算法。仿真实验结果表明，所建立的改进算法的整体性能优于原算法，提升了车辆平均行驶速度，满足最小车头时距同时降低了碰撞率和汇入等待时长，满足了合流区车辆的通行安全和提高了通行效率。

关键词: 强化学习, 合流区, 自动驾驶, 多智能体

LI Chun, WU Zhizhou, XU Hongxin, LIANG Yunyi. Multi-Agent Reinforcement Learning for On-Ramp Merging Control[J]. Computer Engineering and Applications, 2024, 60(23): 349-356.

李春, 吴志周, 许宏鑫, 梁韵逸. 基于多智能体强化学习自动合流控制方法研究[J]. 计算机工程与应用, 2024, 60(23): 349-356.

References

[1] 王景荣, 肖鹤, 解建光. 高速公路拥堵碳排放计算模型研究[J]. 现代交通技术, 2015, 12(2): 81-84.
WANG J R, XIAO H, XIE J G. Research on highway congestion carbon emission calculation model[J]. Modern Transportation Technology, 2015, 12(2): 81-84.
[2] MERGIA W Y, EUSTACE D, CHIMBA D, et al. Exploring factors contributing to injury severity at freeway merging and diverging locations in Ohio[J]. Accident Analysis & Prevention, 2013, 55: 202-210.
[3] 洛玉乐. 匝道合流区网联车辆协同控制策略研究[D]. 哈尔滨: 哈尔滨工业大学, 2022.
LUO Y L. Research on cooperative control strategy of network-connected vehicles in ramp merging area[D]. Harbin: Harbin Institute of Technology, 2022.
[4] WANG H J, GAO H B, YUAN S H, et al. Interpretable decision-making for autonomous vehicles at highway on-ramps with latent space reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2021, 70(9): 8707-8719.
[5] 林祎. 智能网联环境下基于车辆行为优化的匝道管控策略研究[D]. 南京: 东南大学, 2022.
LIN Y. Research on the ramp control strategy based on vehicle behavior optimization in the intelligent networked environment[D]. Nanjing: Southeast University, 2022.
[6] YANG H，RAKHA H. Reinforcement learning ramp metering control for weaving sections in a connected vehicle environment[C]//Proceedings of the 96th Annual Meeting of the Transportation Research Board，2017.
[7] ARAGHI S, KHOSRAVI A, CREIGHTON D. A review on computational intelligence methods for controlling traffic signal timing[J]. Expert Systems with Application, 2015, 42(3): 1538-1550.
[8] 杨建成, 赵武章, 韩自强, 等. 快速路合流区智能网联混合交通组队策略[J]. 武汉理工大学学报 (交通科学与工程版), 2023, 47(3): 440-446.
YANG J C, ZHAO W Z, HAN Z Q, et al. Intelligent network-connected hybrid traffic teaming strategy in expressway merging area[J]. Journal of Wuhan University of Technology (Transportation Science and Engineering Edition), 2023, 47(3): 440-446.
[9] HOU K N, ZHENG F F, LIU X B, et al. Cooperative on-ramp merging control model for mixed traffic on multi-lane freeways[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(10): 10774-10790.
[10] ZHU J, EASA S, GAO K. Merging control strategies of connected and autonomous vehicles at freeway on-ramps: a comprehensive review[J]. Journal of Intelligent and Connected Vehicles, 2022, 5(2): 99-111.
[11] DING J S Y, LI L, PENG H, et al. A rule-based cooperative merging strategy for connected and automated vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(8): 3436-3446.
[12] LETTER C, ELEFTERIADOU L. Efficient control of fully automated connected vehicles at freeway merge segments[J]. Transportation Research Part C: Emerging Technologies, 2017, 20(80): 190-205.
[13] MU C, DU L, ZHAO X M. Event triggered rolling horizon based systematical trajectory planning for merging platoons at mainline-ramp intersection[J]. Transportation Research Part C: Emerging Technologies, 2021, 125: 103006.
[14] HU X, SUN J. Trajectory optimization of connected and autonomous vehicles at a multilane freeway merging area[J]. Transportation Research Part C: Emerging Technologies, 2019, 101: 111-125.
[15] SUMANTH N, CHALAKI B, MALIKOPOULOS A A. Multi-agent deep reinforcement learning coordination framework for connected and automated vehicles at merging roadways[C]//Proceedings of the 2022 American Control Conference, 2022: 3297-3302.
[16] ANTONIO G P, MARIA-DOLORES C. Multi-agent deep reinforcement learning to manage connected autonomous vehicles at tomorrow’s intersections[J]. IEEE Transactions on Vehicular Technology, 2022, 21(7): 7033-7043.
[17] HEUILLET A, COUTHOUIS F, DIAZ-RODRIGUEZ N. Explainability in deep reinforcement learning[J]. Knowledge-Based Systems, 2020, 214: 106685.
[18] LI M, LI Z B, WANG S C, et al. Enhancing cooperation of vehicle merging control in heavy traffic using communication-based soft actor-critic algorithm[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(6): 6491-6506.
[19] LIN Y, MCPHEE J, AZAD N L. Anti-jerk on-ramp merging using deep reinforcement learning[C]//Proeceedings of the 2020 IEEE Intelligent Vehicles Symposium (Ⅳ), 2020: 7-14.
[20] VRBANIC F, IVANJKO E, MANDZUKA S, et al. Reinforcement learning based variable speed limit control for mixed traffic flows[C]//Proceedings of the 2021 29th Mediterranean Conference on Control and Automation, 2021: 560-565.
[21] LIN K X, ZHAO R Y, XU Z, et al. Efficient large-scale fleet management via multi-agent deep reinforcement learning[C]//Proceedings of the 24th ACM International Conference on Knowledge Discovery & Data Mining, 2018: 1774-1783.
[22] ZHUANG H, LEI C, CHEN Y, et al. Cooperative decision-making for mixed traffic at an unsignalized intersection based on multi-agent reinforcement learning[J]. Applied Sciences, 2023, 13(8): 5018.
[23] PARADA L, CANDELA E, MARQUES L, et al. Safe and efficient manoeuvring for emergency vehicles in autonomous traffic using multi-agent proximal policy optimization[J]. Transportmetrica A: Transport Science, 2023: 1-29.
[24] LI L, ZHAO W Z, WANG C Y, et al. Nash double Q-based multi-agent deep reinforcement learning for interactive merging strategy in mixed traffic[J]. Expert Systems with Applications, 2024, 237: 121458.
[25] BAGWE G, YUAN X Y, CHEN X H, et al. RAMRL: towards robust on-ramp merging via augmented multimodal reinforcement learning[C]//Proceedings of the 2023 IEEE International Conference on Mobility, Operations, Services and Technologies, 2023: 23-33.
[26] ZHOU S X, ZHUANG W C, YIN G C, et al. Cooperative on-ramp merging control of connected and automated vehicles: distributed multi-agent deep reinforcement learning approach[C]//Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems, 2022: 402-408.
[27] HENDERSON S G, NELSON B L. Simulation[M]//Handbooks in operations research and management science. Amsterdam: North-Holland, 2006: 331-434.
[28] BOUTON M, NAKHAEI A, FUJIMURA K, et al. Cooperation-aware reinforcement learning for merging in dense traffic[C]//Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference, 2019: 3441-3447.
[29] 孟凡兴, 张良, 张伟. 驾驶员车头时距研究[J]. 工业工程与管理, 2013, 18(2): 131-135.
MENG F X, ZHANG L, ZHANG W. Research on driver headway time distance[J]. Industrial Engineering and Management, 2013, 18(2): 131-135.
[30] LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6382-6393.