{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T17:55:24Z","timestamp":1780595724653,"version":"3.54.1"},"reference-count":40,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T00:00:00Z","timestamp":1675296000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["2022YFB3206800"],"award-info":[{"award-number":["2022YFB3206800"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The rapid development of electric vehicle (EV) technology and the consequent charging demand have brought challenges to the stable operation of distribution networks (DNs). The problem of the collaborative optimization of the charging scheduling of EVs and voltage control of the DN is intractable because the uncertainties of both EVs and the DN need to be considered. In this paper, we propose a deep reinforcement learning (DRL) approach to coordinate EV charging scheduling and distribution network voltage control. The DRL-based strategy contains two layers, the upper layer aims to reduce the operating costs of power generation of distributed generators and power consumption of EVs, and the lower layer controls the Volt\/Var devices to maintain the voltage stability of the distribution network. We model the coordinate EV charging scheduling and voltage control problem in the distribution network as a Markov decision process (MDP). The model considers uncertainties of charging process caused by the charging behavior of EV users, as well as the uncertainty of uncontrollable load, system dynamic electricity price and renewable energy generation. Since the model has a dynamic state space and mixed action outputs, a framework of deep deterministic policy gradient (DDPG) is adopted to train the two-layer agent and the policy network is designed to output discrete and continuous control actions. Simulation and numerical results on the IEEE-33 bus test system demonstrate the effectiveness of the proposed method in collaborative EV charging scheduling and distribution network voltage stabilization.<\/jats:p>","DOI":"10.3390\/s23031618","type":"journal-article","created":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T01:53:54Z","timestamp":1675302834000},"page":"1618","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":59,"title":["Deep Reinforcement Learning for Charging Scheduling of Electric Vehicles Considering Distribution Network Voltage Stability"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3177-9224","authenticated-orcid":false,"given":"Ding","family":"Liu","sequence":"first","affiliation":[{"name":"Key Laboratory of Networked Control Systems, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"University of Chinese Academy of Sciences, Beijing 100049, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peng","family":"Zeng","sequence":"additional","affiliation":[{"name":"Key Laboratory of Networked Control Systems, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shijie","family":"Cui","sequence":"additional","affiliation":[{"name":"Key Laboratory of Networked Control Systems, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8392-1777","authenticated-orcid":false,"given":"Chunhe","family":"Song","sequence":"additional","affiliation":[{"name":"Key Laboratory of Networked Control Systems, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang 110016, China"},{"name":"Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"906","DOI":"10.1109\/TSTE.2016.2617679","article-title":"Optimal Resource Allocation and Charging Prices for Benefit Maximization in Smart PEV-Parking Lots","volume":"8","author":"Awad","year":"2017","journal-title":"IEEE Trans. Sustain. Energy"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"102937","DOI":"10.1016\/j.est.2021.102937","article-title":"Grid integration of battery swapping station: A review","volume":"41","author":"Revankar","year":"2021","journal-title":"J. Energy Storage"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"5246","DOI":"10.1109\/TSG.2018.2879572","article-title":"Model-Free Real-Time EV Charging Scheduling Based on Deep Reinforcement Learning","volume":"10","author":"Wan","year":"2019","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"684","DOI":"10.1109\/JIOT.2021.3084923","article-title":"Smart Online Charging Algorithm for Electric Vehicles via Customized Actor-Critic Learning","volume":"9","author":"Cao","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"13979","DOI":"10.1007\/s13369-022-06624-9","article-title":"Economic Operation Scheduling of Microgrid Integrated with Battery Swapping Station","volume":"47","author":"Revankar","year":"2022","journal-title":"Arab. J. Sci. Eng."},{"key":"ref_6","unstructured":"(2022, October 28). The eGallon: How Much Cheaper Is It to Drive on Electricity?, Available online: https:\/\/www.energy.gov\/articles\/egallon-how-much-cheaper-it-drive-electricity."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MCOM.2016.1600346CM","article-title":"Online Charging Scheduling Algorithms of Electric Vehicles in Smart Grid: An Overview","volume":"54","author":"Tang","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"6883","DOI":"10.1109\/TSG.2019.2913587","article-title":"Smart Control of Fleets of Electric Vehicles in Smart and Connected Communities","volume":"10","author":"Moghaddass","year":"2019","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"13","DOI":"10.35833\/MPCE.2019.000326","article-title":"Grid Integration of Electric Vehicles for Economic Benefits: A Review","volume":"9","author":"Patil","year":"2021","journal-title":"J. Mod. Power Syst. Clean Energy"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2427","DOI":"10.1109\/TSG.2019.2955437","article-title":"Constrained EV Charging Scheduling Based on Safe Deep Reinforcement Learning","volume":"11","author":"Li","year":"2020","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Deng, W., Pei, W., Wu, Q., and Kong, L. (November, January 30). Study on Stability of Low-voltage Multi-terminal DC System Under Electric Vehicle Integration. Proceedings of the 2020 IEEE 4th Conference on Energy Internet and Energy System Integration (EI2), Wuhan, China.","DOI":"10.1109\/EI250167.2020.9347165"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1860","DOI":"10.1109\/TSG.2022.3142961","article-title":"Learning to Operate Distribution Networks With Safe Deep Reinforcement Learning","volume":"13","author":"Li","year":"2022","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4120","DOI":"10.1109\/TPWRS.2020.3000652","article-title":"A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters","volume":"35","author":"Cao","year":"2020","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4873","DOI":"10.1109\/TSG.2022.3185975","article-title":"Multi-agent Deep Reinforcement Learning for Voltage Control with Coordinated Active and Reactive Power Optimization","volume":"13","author":"Hu","year":"2022","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"93352","DOI":"10.1109\/ACCESS.2019.2928173","article-title":"A Support Vector Regression Based Model Predictive Control for Volt-Var Optimization of Distribution Systems","volume":"7","author":"Pourjafari","year":"2019","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"118417","DOI":"10.1109\/ACCESS.2020.3003426","article-title":"A Two-Layer Volt-Var Control Method in Rural Distribution Networks Considering Utilization of Photovoltaic Power","volume":"8","author":"Hu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5711","DOI":"10.1109\/TIA.2022.3183182","article-title":"Two-Stage Volt-VAr Optimization of Distribution Grids With Smart Inverters and Legacy Devices","volume":"58","author":"Savasci","year":"2022","journal-title":"IEEE Trans. Ind. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"5564","DOI":"10.1109\/TSG.2018.2887080","article-title":"Artificial Neural Networks for Volt\/VAR Control of DER Inverters at the Grid Edge","volume":"10","author":"Li","year":"2019","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3008","DOI":"10.1109\/TSG.2019.2962625","article-title":"Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems","volume":"11","author":"Wang","year":"2020","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4752","DOI":"10.1109\/TSG.2021.3094891","article-title":"Hierarchical Voltage Control Strategy in Distribution Networks Considering Customized Charging Navigation of Electric Vehicles","volume":"12","author":"Sun","year":"2021","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"6778","DOI":"10.1109\/TIE.2014.2314065","article-title":"Vehicle-to-Grid Reactive Power Operation Using Plug-In Electric Vehicle Bidirectional Offboard Charger","volume":"61","author":"Kesler","year":"2014","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"638","DOI":"10.1109\/TII.2018.2812755","article-title":"Online Distributed MPC-Based Optimal Scheduling for EV Charging Stations in Distribution Systems","volume":"15","author":"Zheng","year":"2019","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2761","DOI":"10.1109\/TPWRS.2020.3044206","article-title":"Voltage Positioning Using Co-Optimization of Controllable Grid Assets in Radial Networks","volume":"36","author":"Nazir","year":"2021","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2703","DOI":"10.1109\/TSG.2016.2617400","article-title":"Experimental Validation of a Three-Phase Off-Board Electric Vehicle Charger With New Power Grid Voltage Control","volume":"9","author":"Yong","year":"2018","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e12226","DOI":"10.1002\/2050-7038.12226","article-title":"Charging cost minimisation by centralised controlled charging of electric vehicles","volume":"30","author":"Patil","year":"2020","journal-title":"Int. Trans. Electr. Energy Syst."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4229","DOI":"10.1109\/TII.2020.2990397","article-title":"Reinforcement Learning-Based Load Forecasting of Electric Vehicle Charging Station Using Q-Learning Technique","volume":"17","author":"Dabbaghjamanesh","year":"2021","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1173","DOI":"10.1109\/TPWRS.2021.3100994","article-title":"A Novel Cross-Case Electric Vehicle Demand Modeling Based on 3D Convolutional Generative Adversarial Networks","volume":"37","author":"Jahangir","year":"2022","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4738","DOI":"10.1109\/TSG.2020.2998072","article-title":"Plug-in Electric Vehicle Behavior Modeling in Energy Market: A Novel Deep Learning-Based Approach With Clustering Technique","volume":"11","author":"Jahangir","year":"2020","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3284","DOI":"10.1109\/JSYST.2021.3123436","article-title":"Combined Approach for Power Loss Minimization in Distribution Networks in the Presence of Gridable Electric Vehicles and Dispersed Generation","volume":"16","author":"Velamuri","year":"2022","journal-title":"IEEE Syst. J."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2774","DOI":"10.1109\/TSG.2022.3167021","article-title":"EV Charging Strategy Considering Transformer Lifetime via Evolutionary Curriculum Learning-Based Multiagent Deep Reinforcement Learning","volume":"13","author":"Li","year":"2022","journal-title":"IEEE Trans. Smart Grid"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"107912","DOI":"10.1016\/j.ijepes.2021.107912","article-title":"A two-stage joint operation and planning model for sizing and siting of electrical energy storage devices considering demand response programs","volume":"138","author":"Javadi","year":"2022","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_32","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21). Deterministic Policy Gradient Algorithms. Proceedings of the 31st International Conference on Machine Learning, Beijing, China."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_34","unstructured":"Abadi, M.i.N., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2022, August 22). TensorFlow: Large-scale Machine Learning on Heterogeneous Systems. Available online: https:\/\/www.tensorflow.org\/."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6510","DOI":"10.1109\/TPWRS.2018.2829021","article-title":"pandapower\u2014An Open-Source Python Tool for Convenient Modeling, Analysis, and Optimization of Electric Power Systems","volume":"33","author":"Thurner","year":"2018","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1401","DOI":"10.1109\/61.25627","article-title":"Network reconfiguration in distribution systems for loss reduction and load balancing","volume":"4","author":"Baran","year":"1989","journal-title":"IEEE Trans. Power Deliv."},{"key":"ref_37","first-page":"161","article-title":"Real-time Dispatch Strategy for Electric Vehicles Based on Deep Reinforcement Learning","volume":"44","author":"Li","year":"2020","journal-title":"Autom. Electr. Power Syst."},{"key":"ref_38","unstructured":"OASIS (2021, September 09). California ISO Open Access Same-Time Information System. Available online: http:\/\/oasis.caiso.com\/mrioasis\/logon.do."},{"key":"ref_39","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 9). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Proceedings of the International Confernece on Machine Learning, Stockholm, Sweden."},{"key":"ref_40","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithm. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/3\/1618\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:21:39Z","timestamp":1760120499000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/3\/1618"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,2]]},"references-count":40,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23031618"],"URL":"https:\/\/doi.org\/10.3390\/s23031618","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,2]]}}}