基于强化学习的自动驾驶汽车换道决策研究

doi:10.3778/j.issn.1002-8331.2409-0111

摘要/Abstract

摘要： 换道作为车辆行驶的常见行为之一，操作不当极易引发交通事故。针对自动驾驶汽车的换道决策问题，提出了一种基于强化学习的DDQN（双深度Q网络）模型，该模型通过离散动作空间，结合驾驶舒适性、效率、安全性和换道惩罚四个方面设计奖励函数，以优化换道决策。为验证换道决策模型的性能，基于SUMO和真实高速公路车辆数据集搭建高速公路场景下的仿真模型。对比实验结果表明，DDQN模型在驾驶舒适性、交通效率、任务成功率及车辆平均行程速度方面均优于传统的DQN（深度Q网络）和Dueling DQN（对决深度Q网络）模型，且换道次数较少。此外，在四种不同交通拥堵场景下的实验结果显示，DDQN模型在不同拥堵情况下均保持了良好的性能，任务成功率均超过75%。研究表明，基于强化学习的DDQN算法能够为自动驾驶汽车提供有效的换道决策支持。

关键词: 自动驾驶, 换道决策, 强化学习, 马尔可夫决策过程

Abstract: Lane changing, as one of the common behaviors in vehicle driving, can easily lead to traffic accidents if improperly executed. This paper addresses the lane-changing decision-making problem for autonomous vehicles by proposing a double deep Q-network (DDQN) based on reinforcement learning model. The model utilizes a discrete action space and designs a reward function that considers four aspects: driving comfort, efficiency, safety, and lane-changing penalty, aiming to optimize lane-change decisions. To validate the performance of the lane-changing decision model, a simulation model is built for a highway scenario using SUMO and a real highway vehicle dataset. Comparative experimental results show that the DDQN model outperforms traditional DQN and dueling DQN models in terms of driving comfort, traffic efficiency, task success rate, and average vehicle travel speed, with fewer lane changes. Additionally, experimental results under four different traffic congestion scenarios demonstrate that the DDQN model maintains robust performance across varying congestion levels, with task success rates exceeding 75%. This study indicates that the DDQN algorithm based on reinforcement learning can provide effective lane-changing decision support for autonomous vehicles.

Key words: autonomous driving, lane change decision-making, reinforcement learning, Markov decision process

姜文鑫, 吴志周, 许宏鑫, 梁韵逸. 基于强化学习的自动驾驶汽车换道决策研究[J]. 计算机工程与应用, 2025, 61(12): 359-371.

JIANG Wenxin, WU Zhizhou, XU Hongxin, LIANG Yunyi. Research on Lane Change Decision-Making for Autonomous Vehicles Based on Reinforcement Learning[J]. Computer Engineering and Applications, 2025, 61(12): 359-371.

参考文献

[1] 蒲龙忠. 驾驶员驾驶车辆变道行为原因综述[J]. 交通科技与管理, 2021(17): 215.
PU L Z. A summary of the causes of drivers’ lane-changing behavior[J]. Traffic Science and Management, 2021(17): 215.
[2] GIPPS P G. A model for the structure of lane-changing decisions[J]. Transportation Research Part B: Methodological, 1986, 20(5): 403-414.
[3] HIDAS P. Modelling lane changing and merging in microscopic traffic simulation[J]. Transportation Research Part C: Emerging Technologies, 2002, 10(5/6): 351-371.
[4] YU H T, TSENG H E, LANGARI R. A human-like game theory-based controller for automatic lane changing[J]. Transportation Research Part C: Emerging Technologies, 2018, 88: 140-158.
[5] SMIRNOV N, LIU Y Z, VALIDI A, et al. A game theory-based approach for modeling autonomous vehicle behavior in congested, urban lane-changing scenarios[J]. Sensors, 2021, 21(4): 1523.
[6] DONG C Y, WANG H, LI Y, et al. Application of machine learning algorithms in lane-changing model for intelligent vehicles exiting to off-ramp[J]. Transportmetrica A: Transport Science, 2021, 17(1): 124-150.
[7] LIU X, LIANG J, XU B. A deep learning method for lane changing situation assessment and decision making[J]. IEEE Access, 2019, 7: 133749-133759.
[8] 张甫城. 基于机器学习的智能车辆换道决策和轨迹规划研究[D]. 西安: 长安大学, 2021.
ZHANG F C. Research on lane change decisions and trajectory planning of intelligent vehicle through machine learning[D]. Xi’an: Chang’an University, 2021.
[9] WATKINS C J C H. Learning from delayed rewards[D]. Cambridge: University of Cambridge, 1989.
[10] 罗鹏, 黄珍, 秦易晋, 等. 基于DQN的车辆驾驶行为决策方法[J]. 交通信息与安全, 2020, 38(5): 67-77.
LUO P, HUANG Z, QIN Y J, et al. A method of vehicle driving behavior decision based on DQN algorithm[J]. Journal of Transport Information and Safety, 2020, 38(5): 67-77.
[11] 于士杰, 马冲, 陈见哲. 基于学习的自动驾驶换道决策算法研究进展[J]. 汽车实用技术, 2023, 48(24): 189-194.
YU S J, MA C, CHEN J Z. Research progress of automatic driving lane change decision algorithms based on learning[J]. Automobile Applied Technology, 2023, 48(24): 189-194.
[12] 程诺. 基于改进优先经验回放的DDPG路径规划算法研究[D]. 济南: 山东交通学院, 2024.
CHENG N. Research on DDPG path planning algorithm based on improved prioritized experience replay[D]. Jinan: Shandong Jiaotong University, 2024.
[13] 左思翔. 基于深度强化学习的无人驾驶智能决策控制研究[D]. 哈尔滨: 哈尔滨工业大学, 2018.
ZUO S X. Intelligent control of autonomous driving based on deep reinforcement learning[D]. Harbin: Harbin Institute of Technology, 2018.
[14] 张斌, 何明, 陈希亮, 等. 改进DDPG算法在自动驾驶中的应用[J]. 计算机工程与应用, 2019, 55(10): 264-270.
ZHANG B, HE M, CHEN X L, et al. Self-driving via improved DDPG algorithm[J]. Computer Engineering and Applications, 2019, 55(10): 264-270.
[15] 张雪峰, 王照乙. 基于双决斗深度Q网络的自动换道决策模型[J]. 东北大学学报(自然科学版), 2023, 44(10): 1369-1376.
ZHANG X F, WANG Z Y. Automatic lane change decision model based on dueling double deep Q-network[J]. Journal of Northeastern University (Natural Science), 2023, 44(10): 1369-1376.
[16] 裴晓飞, 莫烁杰, 陈祯福, 等. 基于TD3算法的人机混驾交通环境自动驾驶汽车换道研究[J]. 中国公路学报, 2021, 34(11): 246-254.
PEI X F, MO S J, CHEN Z F, et al. Lane changing of autonomous vehicle based on TD3 algorithm in human-machine hybrid driving environment[J]. China Journal of Highway and Transport, 2021, 34(11): 246-254.
[17] 郑施雨. 自动驾驶车辆换道过程建模与分析[D]. 成都: 西南交通大学, 2018.
ZHENG S Y. Modeling and simulation of the lane-changing execution behavior of autonomous vehicle[D]. Chengdu: Southwest Jiaotong University, 2018.
[18] KAELBLING L P, LITTMAN M L, MOORE A W. Reinforcement learning: a survey[J]. Journal of Artificial Intelligence Research, 1996, 4: 237-285.
[19] SUTTON R S, BARTO A G. Reinforcement learning: an introduction[M]. Cambridge MA: MIT Press, 2018.
[20] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533.
[21] WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//Proceedings of the 33rd International Conference on Machine Learning, 2016:1995-2003.
[22] VAN HASSELT, H. Double Q-learning[C]//Proceedings of the 23rd International Conference on Neural Information Processing Systems, 2010: 2613-2621.
[23] HASSELT H V, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2016: 2094-2100.
[24] BEHRISCH M, BIEKER L, ERDMANN J, et al. SUMO-simulation of urban mobility: an overview[C]//Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation, 2011.
[25] 秦严严. 交通流分析理论[M]. 北京: 人民交通出版社, 2023.
QIN Y Y. Theory of traffic flow analysis[M]. Beijing: China Communications Press, 2023.
[26] SUN K, ZHAO X M, WU X. A cooperative lane change model for connected and autonomous vehicles on two lanes highway by considering the traffic efficiency on both lanes[J]. Transportation Research Interdisciplinary Perspectives, 2021, 9: 100310.
[27] DE WINKEL K N, IRMAK T, HAPPEE R, et al. Standards for passenger comfort in automated vehicles: acceleration and jerk[J]. Applied Ergonomics, 2023, 106: 103881.
[28] 中华人民共和国国务院. 中华人民共和国道路交通安全法实施条例[S/OL]. (2019)[2024-08-20]. https://www.gov.cn/gongbao/content/2019/content_5468932.htm.
State Council of the People’s Republic of China. Regulations on the implementation of the road traffic safety law of the People’s Republic of China[S/OL]. (2019)[2024-08-20]. https://www.gov.cn/gongbao/content/2019/content_5468932.htm.
[29] WANG J H, FU T, SHANGGUAN Q Q. Wide-area vehicle trajectory data based on advanced tracking and trajectory splicing technologies: potentials in transportation research[J]. Accident Analysis & Prevention, 2023, 186: 107044.
[30] MO S J, PEI X F, CHEN Z F. Decision-making for oncoming traffic overtaking scenario using double DQN[C]//Proceedings of the 2019 3rd Conference on Vehicle Control and Intelligence. Piscataway: IEEE, 2019: 1-4.
[31] CHEN D, JIANG L S, WANG Y, et al. Autonomous driving using safe reinforcement learning by incorporating a regret-based human lane-changing decision model[C]//Proceedings of the 2020 American Control Conference. Piscataway: IEEE, 2020: 4355-4361.
[32] 中华人民共和国公安部. 道路交通拥堵度评价方法: GA/T 115—2020[S]. 北京: 中国标准出版社, 2020.
Ministry of Public Security of the People’s Republic of China. Evaluation methods for road traffic congestion levels: GA/T 115—2020[S]. Beijing: Standards Press of China, 2020.