{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,19]],"date-time":"2025-10-19T06:08:41Z","timestamp":1760854121939,"version":"build-2065373602"},"reference-count":39,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,1,31]],"date-time":"2020-01-31T00:00:00Z","timestamp":1580428800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61104166"],"award-info":[{"award-number":["61104166"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>In order to solve the problem of vehicle delay caused by stops at signalized intersections, a micro-control method of a left-turning connected and automated vehicle (CAV) based on an improved deep deterministic policy gradient (DDPG) is designed in this paper. In this paper, the micro-control of the whole process of a left-turn vehicle approaching, entering, and leaving a signalized intersection is considered. In addition, in order to solve the problems of low sampling efficiency and overestimation of the critic network of the DDPG algorithm, a positive and negative reward experience replay buffer sampling mechanism and multi-critic network structure are adopted in the DDPG algorithm in this paper. Finally, the effectiveness of the signal control method, six DDPG-based methods (DDPG, PNRERB-1C-DDPG, PNRERB-3C-DDPG, PNRERB-5C-DDPG, PNRERB-5CNG-DDPG, and PNRERB-7C-DDPG), and four DQN-based methods (DQN, Dueling DQN, Double DQN, and Prioritized Replay DQN) are verified under 0.2, 0.5, and 0.7 saturation degrees of left-turning vehicles at a signalized intersection within a VISSIM simulation environment. The results show that the proposed deep reinforcement learning method can get a number of stops benefits ranging from 5% to 94%, stop time benefits ranging from 1% to 99%, and delay benefits ranging from \u221217% to 93%, respectively compared with the traditional signal control method.<\/jats:p>","DOI":"10.3390\/info11020077","type":"journal-article","created":{"date-parts":[[2020,1,31]],"date-time":"2020-01-31T05:55:46Z","timestamp":1580450146000},"page":"77","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Deep Reinforcement Learning Based Left-Turn Connected and Automated Vehicle Control at Signalized Intersection in Vehicle-to-Infrastructure Environment"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0631-8370","authenticated-orcid":false,"given":"Juan","family":"Chen","sequence":"first","affiliation":[{"name":"SILC Business School, Shanghai University, Shanghai 201899, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhengxuan","family":"Xue","sequence":"additional","affiliation":[{"name":"SILC Business School, Shanghai University, Shanghai 201899, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daiqian","family":"Fan","sequence":"additional","affiliation":[{"name":"SILC Business School, Shanghai University, Shanghai 201899, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,1,31]]},"reference":[{"key":"ref_1","first-page":"2","article-title":"Application of artificial intelligence in autonomous vehicles","volume":"3","author":"Zhang","year":"2019","journal-title":"Auto Ind. Res."},{"key":"ref_2","unstructured":"Zhou, B. (China Information World, 2019). 2019 autonomous vehicle maturity index report, China Information World."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.trc.2018.03.008","article-title":"SIV-DSS: Smart In-Vehicle Decision Support System for driving at signalized intersections with V2I communication","volume":"90","author":"Xie","year":"2018","journal-title":"Transp. Res. C"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"822","DOI":"10.1016\/j.trc.2018.09.001","article-title":"Energy saving potentials of connected and automated vehicles","volume":"95","author":"Vahidi","year":"2018","journal-title":"Transp. Res. C"},{"key":"ref_5","first-page":"4","article-title":"Thoughts on the development path of auto insurance in the era of 5G driverless cars","volume":"25","author":"Hou","year":"2019","journal-title":"Technol. Econ. Guide"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1016\/j.trc.2019.05.039","article-title":"Multistep speed prediction on traffic networks: A deep learning approach considering spatio-temporal dependencies","volume":"105","author":"Zhang","year":"2019","journal-title":"Transp. Res. C"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1016\/j.automatica.2019.01.018","article-title":"Data-driven approximate Q-learning stabilization with optimality error bound analysis","volume":"103","author":"Li","year":"2019","journal-title":"Automatica"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"104985","DOI":"10.1016\/j.compag.2019.104985","article-title":"Double-DQN based path smoothing and tracking control method for robotic vehicle navigation","volume":"166","author":"Zhang","year":"2019","journal-title":"Comput. Electron. Agric."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1016\/j.aei.2018.08.002","article-title":"Traffic signal optimization through discrete and continuous reinforcement learning with robustness analysis in downtown Tehran","volume":"38","author":"Aslani","year":"2018","journal-title":"Adv. Eng. Infor."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.procs.2018.04.008","article-title":"Evaluating reinforcement learning state representations for adaptive traffic signal control","volume":"130","author":"Genders","year":"2018","journal-title":"Procedia Comput. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1198","DOI":"10.1109\/TITS.2017.2725912","article-title":"Expert Level control of Ramp Metering based on Multi-task Deep Reinforcement Learning","volume":"19","author":"Belletti","year":"2017","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, P., and Chan, C. (2017, January 16\u201319). Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge. Proceedings of the IEEE 20th International Conference on ITS, Yokohama, Japan.","DOI":"10.1109\/ITSC.2017.8317735"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"70","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023","article-title":"Deep Reinforcement Learning framework for Autonomous Driving","volume":"19","author":"Sallab","year":"2017","journal-title":"Electron. Imag."},{"key":"ref_14","unstructured":"Xu, Y. (2017). Research of Traffic Signal Control in Urban Area based on Cooperative Vehicle Infrastructure System. [Master\u2019s Thesis, North China University of Technology]."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.trc.2014.10.001","article-title":"Signal control optimization for automated vehicles at isolated signalized intersections","volume":"49","author":"Li","year":"2014","journal-title":"Transp. Res. C"},{"key":"ref_16","unstructured":"Sun, X. (2016). Estimation of Emission at Signalized Intersection with Vehicle to Infrastructure Communication. [Master\u2019s Thesis, Beijing Jiaotong University]."},{"key":"ref_17","unstructured":"Guo, Y. (2017). Research and Implementation of Eco-Driving Guidance System for Intersection. [Master\u2019s Thesis, Chang\u2019an University]."},{"key":"ref_18","unstructured":"Wang, X. (2017). Traffic Signal Coordination Control and Optimization for Urban Arterials Considering the Emission of Platoon. [Master\u2019s Thesis, Beijing Jiaotong University]."},{"key":"ref_19","unstructured":"Li, D. (2017). Distinguishing Method of Arterial Fleet Considering Minor Traffic Flow. [Master\u2019s Thesis, Jilin University]."},{"key":"ref_20","unstructured":"Zhou, B. (2016). Urban Traffic Signal Control Based on Connected Vehicle Simulation Platform. [Master\u2019s Thesis, Zhejiang University]."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.trb.2018.04.007","article-title":"Integrated optimization of traffic signals and vehicle trajectories at isolated urban intersections","volume":"112","author":"Yu","year":"2018","journal-title":"Transp. Res. B"},{"key":"ref_22","unstructured":"Liao, R. (2018). Eco-Driving of Vehicle Platoons in Cooperative Vehicle-Infrastructure System at Signalized Intersections. [Master\u2019s Thesis, Beijing Jiaotong University]."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1016\/j.trc.2018.05.025","article-title":"A platoon based cooperative eco-driving model for mixed automated and human-driven vehicles at a signalised intersection","volume":"95","author":"Zhao","year":"2018","journal-title":"Transp. Res. C"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"907","DOI":"10.1016\/j.trd.2018.07.014","article-title":"Eco-driving advisory strategies for a platoon of mixed gasoline and electric vehicles in a connected vehicle system","volume":"63","author":"He","year":"2018","journal-title":"Transp. Res. D"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.trb.2018.07.005","article-title":"Cooperative platoon control for a mixed traffic flow including human drive vehicles and connected and autonomous vehicles","volume":"116","author":"Gong","year":"2018","journal-title":"Transp. Res. B"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kalantari, R., Motro, M., Ghosh, J., and Bhat, C. (2016, January 1\u20134). A distributed, collective intelligence framework for collision-free navigation through busy intersections. Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil.","DOI":"10.1109\/ITSC.2016.7795737"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1177\/0361198118796939","article-title":"Application and Evaluation of the Reinforcement Learning Approach to Eco-Driving at Intersections under Infrastructure-to-Vehicle Communications","volume":"2672","author":"Shi","year":"2018","journal-title":"Transp. Res. Rec."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1016\/j.trpro.2018.12.219","article-title":"Reinforcement Learning of Driver Receiving Traffic Signal Information for Passing through Signalized Intersection at Arterial Road","volume":"37","author":"Matsumoto","year":"2019","journal-title":"Transp. Res. Procedia"},{"key":"ref_29","first-page":"A187","article-title":"Continuous control with deep reinforcement learning","volume":"8","author":"Lillicrap","year":"2015","journal-title":"Comput. Sci."},{"key":"ref_30","unstructured":"Silver, D., Lever, G., Heess, N., and Degris, T. (2014, January 21\u201326). Deterministic Policy Gradient Algorithms. Proceedings of the 31st International Conference on Machine Learning (ICML), Beijing, China."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1016\/S0019-9958(77)90354-0","article-title":"An adaptive optimal controller for discrete-time Markov environments","volume":"34","author":"Witten","year":"1977","journal-title":"Inf. Control"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zuo, S., Wang, Z., Zhu, X., and Ou, Y. (2017, January 5\u20138). Continuous reinforcement learning from human demonstrations with integrated experience replay for autonomous driving. Proceedings of the 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), Macau.","DOI":"10.1109\/ROBIO.2017.8324787"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1016\/j.trc.2018.10.024","article-title":"Human-like autonomous car-following model with deep reinforcement learning","volume":"97","author":"Zhu","year":"2018","journal-title":"Transp. Res. C"},{"key":"ref_34","unstructured":"Xu, J. (2016). Intersection blind spot collision avoidance technology based on Vehicle Infrastructure Cooperative Systems. [Master\u2019s Thesis, Tianjin University of Technology and Education]."},{"key":"ref_35","unstructured":"Jiang, H. (2018). Research on eco-driving control at signalized intersections under connected and automated vehicles environment. [Ph.D. Thesis, Harbin Institute of Technology]."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Sutton, R., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_37","first-page":"1263","article-title":"Reinforcement learning algorithm for partially observable Markov decision processes","volume":"19","author":"Wang","year":"2004","journal-title":"Control Decis."},{"key":"ref_38","unstructured":"Lin, L.J. (1993). Reinforcement Learning for Robots Using Neural Networks, School of Computer Science, Carnegie-Mellon Univ."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wu, J., Wang, R., Li, R., Zhang, H., and Hu, X. (2018, January 7\u201310). Multi-critic DDPG Method and Double Experience Replay. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan.","DOI":"10.1109\/SMC.2018.00039"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/2\/77\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:53:24Z","timestamp":1760172804000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/2\/77"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,31]]},"references-count":39,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["info11020077"],"URL":"https:\/\/doi.org\/10.3390\/info11020077","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2020,1,31]]}}}