{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T17:10:55Z","timestamp":1782321055449,"version":"3.54.5"},"reference-count":52,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T00:00:00Z","timestamp":1719360000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202495"],"award-info":[{"award-number":["52202495"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202494"],"award-info":[{"award-number":["52202494"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Reinforcement Learning (RL) methods are regarded as effective for designing autonomous driving policies. However, even when RL policies are trained to convergence, ensuring their robust safety remains a challenge, particularly in long-tail data. Therefore, decision-making based on RL must adequately consider potential variations in data distribution. This paper presents a framework for highway autonomous driving decisions that prioritizes both safety and robustness. Utilizing the proposed Replay Buffer Constrained Policy Optimization (RECPO) method, this framework updates RL strategies to maximize rewards while ensuring that the policies always remain within safety constraints. We incorporate importance sampling techniques to collect and store data in a Replay buffer during agent operation, allowing the reutilization of data from old policies for training new policy models, thus mitigating potential catastrophic forgetting. Additionally, we transform the highway autonomous driving decision problem into a Constrained Markov Decision Process (CMDP) and apply our proposed RECPO for training, optimizing highway driving policies. Finally, we deploy our method in the CARLA simulation environment and compare its performance in typical highway scenarios against traditional CPO, current advanced strategies based on Deep Deterministic Policy Gradient (DDPG), and IDM + MOBIL (Intelligent Driver Model and the model for minimizing overall braking induced by lane changes). The results show that our framework significantly enhances model convergence speed, safety, and decision-making stability, achieving a zero-collision rate in highway autonomous driving.<\/jats:p>","DOI":"10.3390\/s24134140","type":"journal-article","created":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T09:29:33Z","timestamp":1719394173000},"page":"4140","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Towards Robust Decision-Making for Autonomous Highway Driving Based on Safe Reinforcement Learning"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1597-1961","authenticated-orcid":false,"given":"Rui","family":"Zhao","sequence":"first","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziguo","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8309-7396","authenticated-orcid":false,"given":"Yuze","family":"Fan","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7824-7751","authenticated-orcid":false,"given":"Yun","family":"Li","sequence":"additional","affiliation":[{"name":"Graduate School of Information and Science Technology, The University of Tokyo, Tokyo 113-8654, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4195-5033","authenticated-orcid":false,"given":"Fei","family":"Gao","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Automotive Simulation and Control, Jilin University, Changhun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Cui, G., Zhang, W., Xiao, Y., Yao, L., and Fang, Z. (2022). Cooperative Perception Technology of Autonomous Driving in the Internet of Vehicles Environment: A Review. Sensors, 22.","DOI":"10.3390\/s22155535"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Shan, M., Narula, K., Wong, Y.F., Worrall, S., Khan, M., Alexander, P., and Nebot, E. (2021). Demonstrations of Cooperative Perception: Safety and Robustness in Connected and Automated Vehicle Operations. Sensors, 21.","DOI":"10.3390\/s21010200"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Schiegg, F.A., Llatser, I., Bischoff, D., and Volk, G. (2021). Collective Perception: A Safety Perspective. Sensors, 21.","DOI":"10.3390\/s21010159"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Xiao, W., Mehdipour, N., Collin, A., Bin-Nun, A., Frazzoli, E., Duintjer Tebbens, R., and Belta, C. (2021). Rule-based Optimal Control for Autonomous Driving. arXiv.","DOI":"10.1145\/3450267.3450542"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Collin, A., Bilka, A., Pendleton, S., and Duintjer Tebbens, R. (2021). Safety of the Intended Driving Behavior Using Rulebooks. arXiv.","DOI":"10.1109\/IV47402.2020.9304588"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chen, Y., and Bian, Y. (2023). Tube-based Event-triggered Path Tracking for AUV against Disturbances and Parametric Uncertainties. Electronics, 12.","DOI":"10.3390\/electronics12204248"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Seccamonte, F., Kabzan, J., and Frazzoli, E. (2019). On Maximizing Lateral Clearance of an Autonomous Vehicle in Urban Environments. arXiv.","DOI":"10.1109\/ITSC.2019.8917353"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zheng, L., Yang, R., Peng, Z., Liu, H., Wang, M.Y., and Ma, J. (2023). Real-Time Parallel Trajectory Optimization with Spatiotemporal Safety Constraints for Autonomous Driving in Congested Traffic. arXiv.","DOI":"10.1109\/ITSC57777.2023.10422203"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"11251","DOI":"10.1109\/TVT.2023.3268500","article-title":"Towards Robust Decision-Making for Autonomous Driving on Highway","volume":"72","author":"Yang","year":"2023","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/BF00115009","article-title":"Learning to Predict by the Methods of Temporal Differences","volume":"3","author":"Sutton","year":"1988","journal-title":"Mach. Learn."},{"key":"ref_11","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. arXiv."},{"key":"ref_12","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous Control with Deep Reinforcement Learning. arXiv."},{"key":"ref_13","unstructured":"Li, Y., Li, Y., and Poh, L. (2018). Deep Reinforcement Learning for Autonomous Driving. arXiv."},{"key":"ref_14","unstructured":"Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A.A., Yogamani, S., and P\u00e9rez, P. (2020). Deep Reinforcement Learning for Autonomous Driving: A Survey. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Maramotti, P., Capasso, A.P., Bacchiani, G., and Broggi, A. (2022, January 4\u20139). Tackling real-world autonomous driving using deep reinforcement learning. Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany.","DOI":"10.1109\/IV51971.2022.9827302"},{"key":"ref_16","first-page":"125094","article-title":"An Empirical Study of DDPG and PPO-Based Reinforcement Learning Algorithms for Autonomous Driving","volume":"11","author":"Zhu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5876","DOI":"10.1109\/TVT.2020.2986005","article-title":"A decision-making strategy for vehicle autonomous braking in emergency via deep reinforcement learning","volume":"69","author":"Fu","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hoel, C.-J., Wolff, K., and Laine, L. (2018, January 4\u20137). Automated speed and lane change decision making using deep reinforcement learning. Proceedings of the 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569568"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"512","DOI":"10.1093\/ijpor\/edab001","article-title":"Complementary and competitive framing of driverless cars: Framing effects, attitude volatility, or attitude resistance?","volume":"33","author":"Ho","year":"2021","journal-title":"Int. J. Public Opin. Res."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1109\/TIV.2022.3186897","article-title":"A survey on attack detection and resilience for connected and automated vehicles: From vehicle dynamics and control perspective","volume":"7","author":"Ju","year":"2022","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_21","unstructured":"Tamar, A., Xu, H., and Mannor, S. (2013). Scaling up robust mdps by reinforcement learning. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1613\/jair.1666","article-title":"Risk-sensitive reinforcement learning applied to control under constraints","volume":"24","author":"Geibel","year":"2005","journal-title":"J. Artif. Intell. Res."},{"key":"ref_23","unstructured":"Moldovan, T.M., and Abbeel, P. (2012). Safe exploration in markov decision processes. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1109\/TIV.2022.3170517","article-title":"Formal certification methods for automated vehicle safety assessment","volume":"8","author":"Zhao","year":"2023","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1109\/MITS.2021.3093379","article-title":"On-road object detection and tracking based on radar and vision fusion: A review","volume":"14","author":"Tang","year":"2021","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2460","DOI":"10.1109\/TIE.2020.2973879","article-title":"Path following control of autonomous four-wheel-independent-drive electric vehicles via second-order sliding mode and nonlinear disturbance observer techniques","volume":"68","author":"Chen","year":"2021","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_27","unstructured":"Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained Policy Optimization. arXiv."},{"key":"ref_28","unstructured":"Altman, E. (1999). Constrained Markov Decision Processes: Stochastic Modeling, Routledge."},{"key":"ref_29","unstructured":"Hu, X., Chen, P., Wen, Y., Tang, B., and Chen, L. (2024). Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Driving. arXiv."},{"key":"ref_30","unstructured":"Dulac-Arnold, G., Mankowitz, D.J., and Hester, T. (2019). Challenges of Real-World Reinforcement Learning. arXiv."},{"key":"ref_31","unstructured":"Levine, S., Kumar, V., Tucker, G., and Fu, J. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Bae, S.H., Joo, S.H., Pyo, J.W., Yoon, J.S., Lee, K., and Kuc, T.Y. (2020, January 13\u201316). Finite State Machine based Vehicle System for Autonomous Driving in Urban Environments. Proceedings of the 2020 20th International Conference on Control, Automation and Systems (ICCAS), Busan, Republic of Korea.","DOI":"10.23919\/ICCAS50221.2020.9268341"},{"key":"ref_33","first-page":"124","article-title":"Baidu Apollo EM Motion Planner for Autonomous Driving: Principles, Algorithms, and Performance","volume":"12","author":"Fan","year":"2020","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1002\/rob.20255","article-title":"Autonomous driving in urban environments: Boss and the Urban Challenge","volume":"25","author":"Urmson","year":"2008","journal-title":"J. Field Robot."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Treiber, M., and Kesting, A. (2013). Traffic Flow Dynamics: Data, Models and Simulation, Springer.","DOI":"10.1007\/978-3-642-32460-4"},{"key":"ref_36","first-page":"42","article-title":"Integrating autonomous and assisted driving through a flexible haptic interface","volume":"5","author":"Vanholme","year":"2013","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1002\/rob.20109","article-title":"Using interpolation to improve path planning: The Field D* algorithm","volume":"23","author":"Ferguson","year":"2008","journal-title":"J. Field Robot."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/TIV.2016.2578706","article-title":"A survey of motion planning and control techniques for self-driving urban vehicles","volume":"1","author":"Paden","year":"2016","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Liu, C., Lee, S., Varnhagen, S., and Tseng, H.E. (2017, January 11\u201314). Path planning for autonomous vehicles using model predictive control. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.","DOI":"10.1109\/IVS.2017.7995716"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1002\/rob.20147","article-title":"Stanley: The robot that won the DARPA Grand Challenge","volume":"23","author":"Thrun","year":"2006","journal-title":"J. Field Robot."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"4706","DOI":"10.1109\/TVT.2022.3151651","article-title":"Highway decision-making and motion planning for autonomous driving via soft actor-critic","volume":"71","author":"Tang","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_42","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017, January 13\u201315). CARLA: An Open Urban Driving Simulator. Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, CA, USA."},{"key":"ref_43","unstructured":"Shalev-Shwartz, S., Shammah, S., and Shashua, A. (2016). Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Pan, Y., Cheng, C.-A., Saigol, K., Lee, K., Yan, X., Theodorou, E., and Boots, B. (2017). Agile Autonomous Driving using End-to-End Deep Imitation Learning. arXiv.","DOI":"10.15607\/RSS.2018.XIV.056"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Fulton, N., and Platzer, A. (2018, January 2\u20137). Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12107"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"7419","DOI":"10.1109\/TITS.2021.3069497","article-title":"Confidence-Aware Reinforcement Learning for Self-Driving Cars","volume":"23","author":"Cao","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Tian, R., Sun, L., Bajcsy, A., Tomizuka, M., and Dragan, A.D. (2022, January 23\u201327). Safety assurances for human\u2013robot interaction via confidence-aware game-theoretic human models. Proceedings of the 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, USA.","DOI":"10.1109\/ICRA46639.2022.9812048"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Wen, L., Duan, J., Li, S.E., Xu, S., and Peng, H. (2020). Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization. arXiv.","DOI":"10.1109\/ITSC45102.2020.9294262"},{"key":"ref_49","unstructured":"Xu, H., Zhan, X., and Zhu, X. (March, January 22). Constraints Penalized Q-learning for Safe Offline Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual."},{"key":"ref_50","unstructured":"Zhang, Q., Zhang, L., Xu, H., Shen, L., Wang, B., Chang, Y., Wang, X., Yuan, B., and Tao, D. (2023). SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1805","DOI":"10.1103\/PhysRevE.62.1805","article-title":"Congested Traffic States in Empirical Observations and Microscopic Simulations","volume":"62","author":"Treiber","year":"2000","journal-title":"Phys. Rev. E"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Treiber, M., and Kesting, A. (2009). Modeling lane-changing decisions with MOBIL. Traffic and Granular Flow\u201907, Springer.","DOI":"10.1007\/978-3-540-77074-9_19"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/13\/4140\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:04:37Z","timestamp":1760108677000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/13\/4140"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,26]]},"references-count":52,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["s24134140"],"URL":"https:\/\/doi.org\/10.3390\/s24134140","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,26]]}}}