Review of Connected Autonomous Vehicle Cooperative Control at On-Ramp Merging Areas

doi:10.3778/j.issn.1002-8331.2310-0310

Abstract

Abstract: The area where vehicles conduct interchanges is designated as the on-ramp merging area. The traffic efficiency in the ramp merging area drastically decreases if the mainline and ramp traffic flow density reaches saturation. As a current research hotspot in transportation, intelligent network technology, relying on the high-precision motion control and high-efficiency communication of connected-automated vehicle (CAV), can significantly improve the traffic efficiency in the merging area. The fusion strategies used by CAV are assessed in this research utilizing three different control paradigms: feedback control, optimal control, and reinforcement learning. The shortcomings of the three methods in this scenario are summarized, and specific improvement measures are given by reviewing existing research. Also, it offers a thorough summary of the most recent developments and trends in this particular scientific field.

Key words: on-ramp merging areas, connected and autonomous vehicles, reinforcement learning, optimal control

摘要： 车辆进行交会的区域被指定为上匝道合流区。如果主线和匝道交通流密度达到饱和，匝道合流区的交通效率就会急剧下降。智能网联技术作为当前的交通上的研究热点，依靠智能网联汽车（connected-automated vehicle， CAV）的高精度运动控制和高效率通信，可以显著地提高合流区的通行效率。针对三种不同的控制范式：反馈控制、最优控制和强化学习，对CAV使用的融合策略进行了评估。通过对现有研究的回顾，总结了三种方法在这种情况下的不足之处，并给出了具体的改进措施。此外，全面地总结了这一特定科学领域的最新发展和趋势。

关键词: 匝道合流区, 互联和自动驾驶车辆, 强化学习, 优化控制

LI Chun, WU Zhizhou, ZENG Guang, ZHAO Xin, YANG Zhidan. Review of Connected Autonomous Vehicle Cooperative Control at On-Ramp Merging Areas[J]. Computer Engineering and Applications, 2024, 60(12): 1-17.

李春, 吴志周, 曾广, 赵鑫, 杨志丹. 合流区智能网联汽车协同控制方法综述[J]. 计算机工程与应用, 2024, 60(12): 1-17.

References

[1] BISHOP R. A survey of intelligent vehicle applications worldwide[C]//Proceedings of the IEEE Intelligent Vehicles Symposium, 2000: 25-30.
[2] FARRADYNE P. Vehicle infrastructure integration (VII): architecture and functional requirements[J]. Draft Version, 2005（1）.
[3] SHELL M. Final report of the European safety working group on road safety[EB/OL]. (2003-01) [2023-09-01]. http://www.esafetysupport.info/download/28_recommendations/28_Recommendations.pdf.
[4] 中华人民共和国科技部. 国家高技术研究发展计划 (863计划)现代交通技术领域智能车路协同关键技术研究导则[EB/OL]. (2006-10)[2023-09-01].https://www.most.gov.cn/ztzl/swkjjh/kjjhjj/200610/t20061021_36375.html.
Ministry of Science and Technology of the People’s Republic of China. National high technology research and development program (863 Program) key technology research guidelines for intelligent vehicle-road collaboration in the field of modern transportation technology[EB/OL]. (2006-10) [2023-09-01]. https://www.most.gov.cn/ztzl/swkjjh/kjjhjj/200610/t20061021_
36375.html.
[5] YURTSEVER E, LAMBERT J, CARBALLO A, et al. A survey of autonomous driving: common practices and emerging technologies[J]. IEEE Access, 2020, 8: 58443-58469.
[6] 冉润东.基于深度强化学习的高速公路入口匝道控制方法研究[D].青岛: 山东科技大学, 2019.
RAN R D. Research on freeway ramp metering method based on deep reinforcement learning[D]. Qingdao: Shandong University of Science and Technology, 2019.
[7] ALI Y, ZHENG Z D, HAQUE M M. Modelling lane-changing execution behavior in a connected environment: agrouped random parameters with heterogeneity-in-means approach[J]. Communications in Transportation, 2021, 1: 100009.
[8] 陆海亭, 张宁, 钱振东. 高速道路入口匝道控制方法及应用探索[J]. 公路, 2008(8): 180-186.
LU H T, ZHANG N, QIAN Z D. Discussion about method and application of expressway entrance ramp metering[J]. Highway, 2008(8): 180-186.
[9] 王景荣, 肖鹤, 解建光. 高速公路拥堵碳排放计算模型研究[J]. 现代交通技术, 2015, 12(2): 81-84.
WANG J R, XIAO H, XIE J G. Research on carbon emission calculation model for expressway congestion[J]. Modern Transportation Technology, 2015, 12(2): 81-84.
[10] MERGIA W Y, EUSTACE D, CHIMBA D, et al. Exploring factors contributing to injury severity at freeway merging and diverging locations in Ohio[J]. Accident Analysis & Prevention, 2013, 55: 202-210.
[11] 洛玉乐. 匝道合流区网联车辆协同控制策略研究[D]. 哈尔滨: 哈尔滨工业大学, 2022.
LUO Y L. Research on cooperative controistrategy of connected vehicles in ramp merging area[D]. Harbin: Harbin Institute of Technology, 2021.
[12] 林仪, 揭春雁. 细算 “堵车账” 打通塞车道[N]. 人民政协报, 2016.
LIN Y, JIE C Y. Calculate the “traffic congestion account” to open up the traffic jam[N]. CPPCC Daily, 2016.
[13] 林祎. 智能网联环境下基于车辆行为优化的匝道管控策略研究[D]. 南京: 东南大学, 2022.
LIN Y. Ramp control method based on vehicle behavior optimization under connected and automated vehicles environment[D]. Nanjing: Southeast University, 2022.
[14] 任黎立.高速公路入口匝道控制方法综述[J].交通标准化, 2006(5): 146-149.
REN L L. A review of highway entrance ramp control methods[J]. Traffic Standardization, 2006(5): 146-149.
[15] YANG H, RAKHA H. Reinforcement learning ramp metering control for weaving sections in a connected vehicle environment[C]//Proceedings of the 96th Annual Meeting of the Transportation Research Board, Washington, DC, 2017.
[16] ZHU J, EASA S, GAO K. Merging control strategies of connected and autonomous vehicles at freeway on-ramps: a comprehensive review[J]. Journal of Intelligent and Connected Vehicles, 2022, 3(1): 1-23.
[17] PUEBOOBPAPHAN R, LIU F, AREM B V. The impacts of a communication based merging assistant on traffic flows of manual and equipped vehicles at an on-ramp using traffic flow simulation[C]//Proceedings of the 13th International IEEE Conference on Intelligent Transportation Systems, 2010: 1468-1473.
[18] RAN B, LEIGHT S, CHANG B. A microscopic simulation model for merging control on a dedicated-lane automated highway system[J]. Transportation Research Part C: Emerging Technologies, 1999, 7(6): 369-388.
[19] WANG Z, KULIK L, RAMAMOHANARAO K. Proactive traffic merging strategies for sensor-enabled cars[C]//Proceedings of the 4th ACM International Workshop on Vehicular Ad Hoc Networks, 2007: 39-48.
[20] DAAMEN W, AREM B V, BOUMA I. Microscopic dynamic traffic management: simulation of two typical situations[C]//Proceedings of the 14th International IEEE Conference on Intelligent Transportation Systems, 2011: 1898-1903.
[21] AWAL T, KULIK L, RAMAMOHANRAO K. Optimal traffic merging strategy for communication-and sensorenabled vehicles[C]//Proceedings of the 16th International IEEE Conference on Intelligent Transportation Systems, 2013: 1468-1474.
[22] XIE Y C, ZHANG H X, GARTNER N H, et al. Collaborative merging strategy for freeway ramp operations in a connected and autonomous vehicles environment[J]. Journal of Intelligent Transportation Systems, 2016, 21(2): 136-147.
[23] XU L, LU J, RAN B, et al. Cooperative merging strategy for connected vehicles at highway on-ramps[J]. Journal of Transportation Engineering, 2019, 145(6): 04019022.
[24] FANG Y K, MIN H G, WU X, et al. On-ramp merging strategies of connected and automated vehicles considering communication delay[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(9): 15298-15312.
[25] ZHAO C, CHU D, WANG R, et al. Consensus control of highway on-ramp merging with communication delays[J]. IEEE Transactions on Vehicular Technology, 2022, 71(9): 9127-9142.
[26] TONG X L, SHI Y, ZHANG Q Y, et al. Adaptive on-ramp merging strategy under imperfect communication performance[J]. Vehicular Communications, 2023, 44: 2096-2214.
[27] UNO A, SAKAGUCHI T, TSUGAWA S. A merging control algorithm based on inter-vehicle communication[C]//Proceedings of the IEEE/IEEJ/JSAI International Conference on Intelligent Transportation Systems, 1999: 783-787.
[28] SAKAGUCHI T, UNO A, KATO S, et al. Cooperative driving of automated vehicles with inter-vehicle communications[C]//Proceedings of the 2000 IEEE Intelligent Vehicles Symposium, 2000: 516-521.
[29] XU Q, SENGUPTA R. Simulation, analysis, and comparison of ACC and CACC in highway merging control[C]//Proceedings of the IEEE IV 2003 Intelligent Vehicles Symposium, 2003: 237-242.
[30] CHOU F C, SHLADOVER S E, BANSAL G. Coordinated merge control based on V2V communication[C]//Proceedings of the 2016 IEEE Vehicular Networking Conference (VNC), 2016: 1-8.
[31] LU X Y, TAN H S, SHLADOVER S E, et al. Modeling, design and implementation of longitudinal control algorithm for automated vehicle merging[J]. California PATH Research Report, 2000(9).
[32] LU X Y, TAN H S, SHLADOVER S E, et al. Automated vehicle merging maneuver implementation for AHS[J]. Vehicle System Dynamics, 2004, 41(2): 85-107.
[33] HUANG Z, ZHUANG W, YIN G, et al. Cooperative merging for multiple connected and automated vehicles at highway on-ramps via virtual platoon formation[C]//Proceedings of the 2019 Chinese Control Conference (CCC), 2019: 6709-6714.
[34] MORLA R. Vision of congestion-free road traffic and cooperating objects[EB/OL]. (2005-01)[2023-09-01]. https://api.semanticscholar.org/CorpusID:18480990.
[35] MARINESCU D, ?URN J, SLOT M, et al. An active approach to guaranteed arrival times based on traffic shaping[C]//Proceedings of the 13th International IEEE Conference on Intelligent Transportation Systems, 2010: 1711-1717.
[36] MARINESCU D, ?URN J, BOUROCHE M, et al. On-ramp traffic merging using cooperative intelligent vehicles: a slot-based approach[C]//Proceedings of the 15th International IEEE Conference on Intelligent Transportation Systems, 2012: 900-906.
[37] MENG T C, HU Z Y, JIN H, et al. Collision-free control strategy for on-ramp merging: a spatial-dependent constraint following approach[J]. Journal of Physics: Conference Series, 2022, 2234: 012011.
[38] 赵红专, 陈智振, 代静. 一种基于双层模糊控制的匝道合流区协同控制方法[J]. 大众科技, 2022, 24(12): 9-13.
ZHAO H Z, CHEN Z Z, DAI J. A coordinated control method of ramp confluence area based on two-layer fuzzy control[J]. Popular Science & Technology, 2022, 24(12): 9-13.
[39] MILANES V, GODOY J, VILLAGRA J, et al. Automated on-ramp merging system for congested traffic situations[J]. IEEE Transactions on Intelligent Transportation Systems, 2011, 12(2): 500-508.
[40] CHIANG T C, WANG W J. Highway on-ramp control using fuzzy decision making[J]. Journal of Vibration and Control, 2011, 17(2): 205-213.
[41] XU J, ZHAO X, SRINIVASAN D. On optimal freeway local ramp metering using fuzzy logic control with particle swarm optimization[J]. Intelligent Transport Systems, 2013, 7: 95-104.
[42] HUANG C P, JIANG M, CHAI G. Fuzzy control for ramp metering and variable speed limitation of freeway[J]. Computer Technology and Development, 2010, 20(12): 38-41.
[43] LIANG X, LI Z. Freeway ramp control based on genetic PI and fuzzy logic[C]//Proceedings of IEEE Pacific-Asia Workshop on Computational Intelligence and Industrial Application, Wuhan, China, 2008: 382-387.
[44] XU J, ZHAO X, SRINIVASAN D. Fuzzy logic controller for freeway ramp metering with particle swarm optimization and PARAMICS simulation[C]//Proceedings of International Conference on Fuzzy Systems, Barcelona, Spain, 2010: 1-6.
[45] RAMOS G, PINTO M, COELHO F, et al. Hybrid methodology based on computational vision and sensor fusion for assisting autonomous UAV on offshore messenger cable transfer operation[J]. Robotica, 2022, 40(8): 2786-2814.
[46] LI A, CAO J, LI S, et al. Map construction and path planning method for a mobile robot based on multi-sensor information fusion[J]. Applied Sciences, 2022, 12: 2913.
[47] HAIDER M H, WANG Z L, KHAN A A, et al. Robust mobile robot navigation in cluttered environments based on hybrid adaptive neuro-fuzzy inference and sensor fusion[J]. Journal of King Saud University (Computer and Information Sciences), 2022, 34(10): 1319-1578.
[48] SONG R, HEGDE A, SENEL N, et al. Edge-aided sensor data sharing in vehicular communication networks[C]//Proceedings of the IEEE 95th Vehicular Technology Conference (VTC2022-Spring), Helsinki, Finland, 2022: 1-7.
[49] KOSURU V S R, VENKITARAMAN A K. A smart battery management system for electric vehicles using deep learning-based sensor fault detection[J].?World Electric Vehicle Journal, 2023, 14: 101.
[50] FANG Y, MIN H, LEI X, et al. A self-fault diagnosis framework for sensors of connected and automated vehicles with dynamic environmental impact quantification[C]//Proceedings of the IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022: 422-442.
[51] ANDRADE E, MATOS F, SANTOS A. A virtual models-based CAVs platoon resilient to network and sensor attacks[J]. Ad Hoc Networks, 2023, 149: 103225.
[52] MOUSAVI S S, BAHRAMI S, KOUVELAS A. Controller design for a mixed traffic system travelling at different desired speeds[J]. European Journal of Control, 2022, 68: 100698.
[53] HUA Z, ZHAN J, ZHANG L. Decentralized tube model predictive control for arbitrarily mixed vehicle platoons with HDV uncertainties[C]//Proceedings of the 41st Chinese Control Conference, Hefei, China, 2022: 5429-5434.
[54] SINGH N K, SAHA I. STL-based synthesis of feedback controllers using reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2023: 15118-15126.
[55] CHEN M, GUO J J, WEN C K, et al. Deep learning-based implicit CSI feedback in massive MIMO[J]. IEEE Transactions on Communications, 2022, 70(2): 935-950.
[56] GONZáLEZ G W, MONTOYA O D. Active and reactive power conditioning using SMES devices with PMW-CSC: a feedback nonlinear control approach[J]. Ain Shams Engineering Journal, 2019, 10(2): 369-378.
[57] AVEDISOV S S, HE C R, TAKáCS D, et al. Machine learning-based steering control for automated vehicles utilizing V2X communication[C]//Proceedings of the 2021 IEEE Conference on Control Technology and Applications (CCTA), San Diego, CA, USA, 2021: 253-258.
[58] VALIENTE R, RAFTARI A, MAHJOUB H N. Context-aware target classification with hybrid Gaussian process prediction for cooperative vehicle safety systems[J]. arXiv: 2212.12819, 2022.
[59] ZHANG E, MASOUD N, BANDEGI M, et al. Predicting risky driving in a connected vehicle environment[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10): 17177-17188.
[60] ZHANG H, RITCHIE S G, RECKER W W. Some general results on the optimal ramp control problem[J]. Transportation Research Part C, 1996, 4(2): 51-69.
[61] YOSHINO T, SASAKI T, HASEGAWA T.?The traffic-control system on the hanshin expressway[J]. Interfaces, 1995, 25(1): 94-108.
[62] HEGYI A, SCHUTTER B D, HEELENDOORN J. MPC-based optimal coordination of variable speed limits to suppress shock waves in freeway traffic[C]//Proceedings of the 2003 American Control Conference, 2003: 4083-4088.
[63] LI L, WANG F. Cooperative driving at blind crossings using intervehicle communication[J]. IEEE Transactions on Vehicular Technology, 2006, 55(6):1712-1724.
[64] MULLER E, CARLSON R, JUNIOR W. Intersection control for automated vehicles with MILP[J]. IFAC-Papers OnLine, 2016, 49(3):37-42.
[65] FAYAZI S, VAHIDI A, LUCKOW A. Optimal scheduling of autonomous vehicle arrivals at intelligent intersections via MILP[C]//Proceedings of the 2017 American Control Conference (ACC), 2017: 4920-4925.
[66] RIOS J, MALIKOPOULOS A, PISU P. Online optimal control of connected vehicles for efficient traffic flow at merging roads[C]//Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, 2015: 2432-2437.
[67] NTOUSAKIS A, NIKOLOS I, PAPAGEORGIOU M. Optimal vehicle trajectory planning in the context of cooperative merging on highways[J]. Transportation Research Part C: Emerging Technologies, 2016, 71:464-488.
[68] JIN X F, YU X H, HU Y H, et al. Integrated control of internal boundary and ramp inflows for lane-free traffic of automated vehicles on freeways[C]//Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022:1234-1239.
[69] VISHNOI S C, JI J Y. CAV Traffic control to mitigate the impact of congestion from bottlenecks: a linear quadratic regulator approach and microsimulation study[J]. arXiv:2306.10418, 2023.
[70] ZHANG H M, RECKER W W. On optimal freeway ramp control policies for congested traffic corridors[J]. Transportation Research Part B: Methodological, 1999, 3370(6):417-436.
[71] KOTSIALOS A, PAPAGEORGIOU M, MIDDELHAM F. Optimal coordinated ramp metering with advanced motorway optimal control[J]. Transportation Research Record Journal of the Transportation Research Board, 2001, 1748:55-65.
[72] TIAN L P, ZHU H G, ZHU X D, et al. A vehicle-road cooperative speed limit method for highway ramp merge zone traffic efficiency[J]. Highway, 2019, 64(8): 310-316.
[73] CHEN D, MO F, CHEN Y, et al. Optimization of ramp locations along freeways: a dynamic programming approach[J]. Sustainability, 2022, 14: 9718.
[74] PEI H, FENG S, ZHANG Y, et al. A cooperative driving strategy for merging at on-ramps based on dynamic programming[J]. IEEE Transactions on Vehicular Technology, 2019, 68(12): 11646-11656.
[75] AHN H, VECCHIO D. Safety verification and control for collision avoidance at road intersections[J]. IEEE Transactions on Automatic Control, 2018, 63(3):630-642.
[76] ZOHDY I, RAKHA H. Intersection management via vehicle connectivity: the intersection cooperative adaptive cruise control system concept[J]. Journal of Intelligent Transportation Systems, 2014, 20(1):17-32.
[77] BELLEMANS T, SCHUTTER B D, MOOR B D. Model predictive control with repeated model fitting for ramp metering[C]//Proceedings of the IEEE 5th International Conference on Intelligent Transportation Systems, 2002: 236-241.
[78] ZHOU Y, CHOLETTE M, BHASKAR A, et al. Optimal vehicle trajectory planning with control constraints and recursive implementation for automated on-ramp merging[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(9): 3409-3420.
[79] MALIKOPOULOS A, LIU Z. A closed-form analytical solution for optimal coordination of connected and automated vehicles[C]//Proceedings of the 2019 American Control Conference (ACC), 2019: 3599-3604.
[80] MALIKOPOULOS A, CASSANDRAS C, ZHANG Y. A decentralized energy-optimal control framework for connected automated vehicles at signal-free intersections[J]. Automatica, 2018, 93: 244-256.
[81] XIAO W, CASSANDRAS C. Decentralized optimal merging control for connected and automated vehicles with safety constraint guarantees[J]. Automatica, 2021, 123: 109333.
[82] ZHANG Y, CASSANDRAS C, MALIKOPOULOS A. Optimal control of connected and automated vehicles at urban traffic intersections: a feasibility enforcement analysis[C]//Proceedings of the 2017 American Control Conference (ACC), 2017: 3548-3553.
[83] CAO W, MUKAI M, KAWABE K, et al. Cooperative vehicle path generation during merging using model predictive control with real-time optimization[J]. Control Engineering Practice, 2015, 34: 98-105.
[84] XIAO X, CASSANDRAS C, BELTA C. Decentralized optimal control in multi-lane merging for connected and automated vehicles[C]//Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020: 1-6.
[85] IOANNIS A, NTOUSAKIS, IOANNIS K, et al. Optimal vehicle trajectory planning in the context of cooperative merging on highways[J]. Transportation Research Part C: Emerging Technologies, 2016, 71: 464-488.
[86] XIAO W, CASSANDRAS C G, BELTA C. Decentralized Optimal control in multi-lane merging for connected and automated vehicles[C]//Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 2020: 1-6.
[87] LIU J, ZHAO W, XU C. An efficient on-ramp merging strategy for connected and automated vehicles in multi-lane traffic[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(6): 5056-5067.
[88] ZHOU Y, CHUNG E, BHASKAR A, et al. A state-constrained optimal control based trajectory planning strategy for cooperative freeway mainline facilitating and on-ramp merging maneuvers under congested traffic[J]. Transportation Research Part C: Emerging Technologies, 2019, 109: 321-342.
[89] 柳祖鹏, 黄镓法, 王道斌. 基于合作博弈的合流区加速路段智能网联汽车协同控制[J].交通运输研究, 2023, 9(6): 34-43.
LIU Z P, HUANG J F, WANG D B. Collaborative control of inteligent connected vehicles on acceleration section of merging area based on cooperative game[J]. Transport Research, 2023, 9(6):34-43.
[90] GAO Z, WU Z, HAO W, et al. Optimal trajectory planning of connected and automated vehicles at on-ramp merging area[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8): 12675-12687.
[91] LIANG J, GUAN T, LIU D, et al. An optimal trajectory planning for automated on-ramp merging[J]. IET Intelligent Transport Systems, 2023, 17: 835-847.
[92] XUE Y J, ZHANG X K, CUI Z Y, et al. A platoon-based cooperative optimal control for connected autonomous vehicles at highway on-ramps under heavy traffic[J]. Transportation Research Part C: Emerging Technologies, 2023, 150:104083.
[93] JING S, HUI F, ZHAO X, et al. Integrated longitudinal and lateral hierarchical control of cooperative merging of connected and automated vehicles at on-ramps[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(12): 24248-24262.
[94] LIU C, ZHENG F F, LI R J, et al. Robust optimal control of connected automated vehicles considering dynamic platoon size in mixed traffic environment[J]. SSRN Electronic Journal, 2022(1).
[95] MA G Q, WANG B Q, GE S S. Robust optimal control of connected and automated vehicle platoons through improved particle swarm optimization[J]. Transportation Research Part C: Emerging Technologies, 2022, 135: 103488.
[96] HUANG M, JIANG Z P, OZBAY K. Learning-based adaptive optimal control for connected vehicles in mixed traffic: robustness to driver reaction time[J]. IEEE Transactions on Cybernetics, 2022, 52(6): 5267-5277.
[97] OZAKI N, CAMPAGNOL S, FUNASE R. Tube stochastic optimal control for nonlinear constrained trajectory optimization problems[J]. Journal of Guidance, Control, and Dynamics, 2020, 43(4): 645-655.
[98] MALANOWSKI K, BüSKENS C, MAURER H. Convergence of approximations to nonlinear optimal control problems[J]. Mathematical Programming with Data Perturbations, 2020(9): 253-284.
[99] MERKT W X, IVAN V, DINEV T, et al. Memory clustering using persistent homology for multimodality-and discontinuity-sensitive learning of optimal control warm-starts[J]. IEEE Transactions on Robotics, 2021, 37(5): 1649-1660.
[100] TANG G, HAUSER K. A data-driven indirect method for nonlinear optimal control[J]. Astrodyn, 2019, 4(3): 345-359.
[101] FURUKAWA J I, MORIMOTO J. Composing an assistive control strategy based on linear bellman combination from estimated user’s motor goal[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 1051-1058.
[102] ZHAO J, LV Y, ZENG Q, et al. Online policy learning based output-feedback optimal control of continuous-time systems[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2022, 71(2): 652-656.
[103] JI Q L, CHEN M, WANG X V, et al. Optimal shape morphing control of 4D printed shape memory polymer based on reinforcement learning[J]. Robotics and Computer-Integrated Manufacturing, 2022, 73: 102209.
[104] WU Y, ZHANG J, SHEN T. A logical network approximation to optimal control on a continuous domain and its application to HEV control[J]. Science China Information Sciences, 2022, 65: 212203.
[105] BESTEHORN F, KIRCHES C. The integrated control deviation of mixed-integer optimal control problems with vanishing constraints[J]. Proceedings in Applied Mathematics and Mechanics, 2021, 20: e202000022.
[106] HAN J H, VAHIDI A. Antonio sciarretta,fundamentals of energy efficient driving for combustion engine and electric vehicles: an optimal control perspective[J]. Automatica, 2019, 103: 558-572.
[107] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518: 529-533.
[108] HAYDARI A, YILMAZ Y. Deep reinforcement learning for intelligent transportation systems: a survey[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(1): 11-32.
[109] LIN L J. Reinforcement learning for robots using neural networks[D]. Pennsylvania: Carnegie Mellon University, 1993.
[110] QIAO L, BAO H, XUAN Z X, et al. Driverless ramp convergence model based on reinforcement earning[J]. Computer Engineering, 2018, 44(7): 2024-2031.
[111] WANG H J, GAO H B, YUAN S H, et al. Interpretable decision-making for autonomous vehicles at highway on-ramps with latent space reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2021, 70(9): 8707-8719.
[112] TOMOKI N, PRASHANT D, DANIL V. Prokhorov merging in congested freeway traffic using multipolicy decision making and passive actor-critic learning[J]. IEEE Transactions on Intelligent Vehicles, 2019, 4(2): 287-297.
[113] WANG P, CHAN C. Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge[C]//Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems, Yokohama, Japan, 2017: 1-6.
[114] LIU J, ZHAO W, XU C. An efficient on-ramp merging strategy for connected and automated vehicles in multi-lane traffic[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(6): 5056-5067.
[115] LIN Y, MCPHEE J, AZAD N. Anti-jerk on-ramp merging using deep reinforcement learning[C]//Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), 2020:7-14.
[116] HU Y, NAKHAEI A, TOMIZUKA M, et al. Interaction-aware decision making with adaptive strategies under merging scenarios[C]//Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019: 151-158.
[117] CHEN T Y, WANG M, GONG S Y, et al. Connected and automated vehicle distributed control for on-ramp merging scenario: a virtual rotation approach[J]. Transportation Research Part C: Emerging Technologies, 2021, 133:103451.
[118] ANDERSSON O, HEINTZ F, DOHERTY P. Model-based reinforcement learning in continuous environments using real-time constrained optimization[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2015: 2497-2503.
[119] KAMTHE S, DEISENROTH M P. Data-efficient reinforcement learning with probabilistic model predictive control[J]. arXiv:1706.06491,2017.
[120] LEE G, SRINIVASA S S, MASON M Y. Data-driven robust optimal control for uncertain nonlinear dynamical systems[J]. arXiv:1705.05344, 2017.
[121] DEISENROTH M P, FOX D, RASMUSSEN C E. Gaussian processes for data-efficient learning in robotics and control[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(2): 408-423.
[122] LUO F M, XU T, LAI H, et al. A survey on model-based reinforcement learning[J]. arXiv:2206.09328, 2022.
[123] LEE H, KIM K, KIM N, et al. Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning[J]. Applied Energy, 2022, 313: 118460.
[124] NIEKERK B V, DAMIANOU A, ROSMAN B. Online constrained model-based reinforcement learning[J]. arXiv:2004.03499,2020.
[125] ZHANG W, WANG J X, XU Z Y. A generalized energy management framework for hybrid construction vehicles via model-based reinforcement learning[J]. Energy, 2022, 260:124849.
[126] ZHOU S, ZHUANG W, YIN G, et al. Cooperative on-ramp merging control of connected and automated vehicles: distributed multi-agent deep reinforcement learning approach[C]//Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022: 402-408.
[127] MAHATTHANAJATUPHAT C, SRISOMBOON K, LEE W, et al. Investigation of multi-agent reinforcement learning on merge ramp for avoiding car crash on highway[C]//Proceedings of the 2022 37th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), Phuket, Thailand, 2022: 1050-1053.
[128] LE N. Multi-agent reinforcement learning for traffic congestion on one-way multi-lane highways[J]. Journal of Information and Telecommunication, 2023, 14(6): 348.
[129] LIU Q, LI Z, LI X, et al. Graph convolution-based deep reinforcement learning for multi-agent decision-making in interactive traffic scenarios[C]//Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China, 2022: 4074-4081.
[130] ZHANG H, ZHANG S. Multi-agent reinforcement learning[J]. Deep Reinforcement Learning, 2020 (1): 335-346.
[131] AJAO L A, APEH S T. Secure edge computing vulnerabilities in smart cities sustainability using petri net and genetic algorithm-based reinforcement learning[J]. Intelligent Systems with Applications, 2023, 18: 200216.
[132] CHEN H Y, ZHANG Y, BHATTI U A, et al. Safe decision controller for autonomous drivingbased on deep reinforcement learning in nondeterministic environment[J]. Sensors, 2023, 23: 1198.
[133] HARRIS A, VALADE T, TEIL T, et al. Generation of spacecraft operations procedures using deep reinforcement learning[J]. Journal of Spacecraft and Rockets, 2022, 59(2): 611-626.
[134] HU J, WANG L, HU T, et al. Autonomous maneuver decision making of dual-UAV cooperative air combat based on deep reinforcement learning[J]. Electronics, 2022, 11: 467.
[135] JIN C, YANG Z R, WANG Z R, et al. Provably efficient reinforcement learning with linear function approximation[J]. arXiv:1907.05388, 2019.
[136] HU P, CHEN Y, HUANG L B. Nearly minimax optimal reinforcement learning with linear function approximation[J]. arXiv:2206.11489, 2022.
[137] ZHAN G, ZHANG X, LI Z, et al. Multiple-UAV reinforcement learning algorithm based on improved ppo in ray framework[J]. Drones, 2022, 6: 166.
[138] LIU B, DING Z. A distributed deep reinforcement learning method for traffic light control[J]. Neurocomputing, 2022, 490: 390-399.
[139] WANG H C, HUANG S C, HUANG P J, et al. Curriculum reinforcement learning from avoiding collisions to navigating among movable obstacles in diverse environments[J]. IEEE Robotics and Automation Letters, 2023, 8(5): 2740-2747.
[140] WANG B, XIE J, ATANASOV N. DARL1N: distributed multi-agent reinforcement learning with one-hop neighbors[C]//Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022: 9003-9010.
[141] FU Y, LI C, YU F R, et al. A selective federated reinforcement learning strategy for autonomous driving[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(2): 1655-1668.
[142] LUO S, KASAEI H, SCHOMAKER L. Accelerating reinforcement learning for reaching using continuous curriculum learning[C]//Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, 2020: 1-8.
[143] SONG Y, LIN H, KAUFMANN E, et al. Autonomous overtaking in gran turismo sport using curriculum reinforcement learning[C]//Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 2021: 9403-9409.
[144] GUPTA K, MUKHERJEE D, NAJJARAN H. Extending the capabilities of reinforcement learning through curriculum: a review of methods and applications[J]. SN Computer Science, 2022, 3(1): 28.
[145] HUANG Z, WU J, LV C. Efficient deep reinforcement learning with imitative expert priors for autonomous driving[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(10): 7391-7403.
[146] 章程, 赵靖, 杨晓光, 等. 城市快速路入口匝道控制研究综述[J]. 上海理工大学学报, 2023, 45(4): 332-344.
ZHANG C, ZHAO J, YANG X G, et al. A review of ramp metering for urban expressway[J]. Journal of University of Shanghai for Science and Technology, 2023, 45(4): 332-344.