{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T06:14:02Z","timestamp":1778134442569,"version":"3.51.4"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"S1","license":[{"start":{"date-parts":[[2022,9,7]],"date-time":"2022-09-07T00:00:00Z","timestamp":1662508800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,7]],"date-time":"2022-09-07T00:00:00Z","timestamp":1662508800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Energy Inform"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Electrification of the transportation sector provides several advantages in favor of climate protection and a shared economy. At the same time, the rapid growth of electric vehicles also demands innovative solutions to mitigate risks to the low-voltage network due to unpredictable charging patterns of electric vehicles. This article conceptualizes a stochastic reinforcement learning agent that learns the optimal policy for regulating the charging power. The optimization objective intends to reduce charging time, thus charging faster while minimizing the expected voltage violations in the distribution network. The problem is formulated as a two-stage optimization routine where the stochastic policy gradient agent predicts the boundary condition of the inner non-linear optimization problem. The results confirm the performance of the proposed architecture to control the charging power as intended. The article also provides extensive theoretical background and directions for future research in this discipline.<\/jats:p>","DOI":"10.1186\/s42162-022-00197-5","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T14:03:07Z","timestamp":1662645787000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A stochastic deep reinforcement learning agent for grid-friendly electric vehicle charging management"],"prefix":"10.1186","volume":"5","author":[{"given":"Charitha Buddhika","family":"Heendeniya","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lorenzo","family":"Nespoli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,9,7]]},"reference":[{"key":"197_CR1","doi-asserted-by":"publisher","first-page":"41506","DOI":"10.1109\/ACCESS.2021.3064354","volume":"9","author":"HM Abdullah","year":"2021","unstructured":"Abdullah HM, Gastli A, Ben-Brahim L (2021) Reinforcement learning based EV charging management systems\u2014a review. IEEE Access 9:41506\u201341531","journal-title":"IEEE Access"},{"key":"197_CR2","unstructured":"Afjei T, Schonhardt U, Wemh\u00f6ner C, Erb M, Gabathuler HR, Mayer H, Zweifel G, Achermann M, von Euw R, St\u00f6ckli U (2002) Standardschaltungen f\u00fcr Kleinw\u00e4rmepumpenanlagen Teil 2: Grundlagen und Computersimulationen. Schlussbericht, Technical report"},{"issue":"4","key":"197_CR3","doi-asserted-by":"publisher","first-page":"2449","DOI":"10.3390\/en7042449","volume":"7","author":"M Alonso","year":"2014","unstructured":"Alonso M, Amaris H, Germain JG, Galan JM (2014) Optimal charging scheduling of electric vehicles in smart grids by heuristic algorithms. Energies 7(4):2449\u20132475","journal-title":"Energies"},{"key":"197_CR4","doi-asserted-by":"crossref","unstructured":"Ayaz MS, Azizipanah-Abarghooee R, Terzija V (2018) European LV microgrid benchmark network: Development and frequency response analysis. 2018 IEEE International Energy Conference, ENERGYCON 2018, 1\u20136","DOI":"10.1109\/ENERGYCON.2018.8398759"},{"issue":"8","key":"197_CR5","doi-asserted-by":"publisher","first-page":"106","DOI":"10.3390\/pr6080106","volume":"6","author":"L Beal","year":"2018","unstructured":"Beal L, Hill D, Martin R, Hedengren J (2018) Gekko optimization suite. Processes 6(8):106","journal-title":"Processes"},{"key":"197_CR6","doi-asserted-by":"crossref","unstructured":"Bla\u017ei\u010d B, Papi\u010d I (2008) Voltage profile support in distribution networks - Influence of the network R\/X ratio. 2008 13th International Power Electronics and Motion Control Conference, EPE-PEMC 2008, 2510\u20132515","DOI":"10.1109\/EPEPEMC.2008.4635641"},{"key":"197_CR7","unstructured":"Blum C, Liu H, Xiong H (2021) CoordiQ: Coordinated Q-learning for Electric Vehicle Charging Recommendation"},{"key":"197_CR8","doi-asserted-by":"crossref","unstructured":"Bucher C, Betcke J, Andersson G (2013) Effects of variation of temporal resolution on domestic power and solar irradiance measurements. 2013 IEEE Grenoble Conference PowerTech, POWERTECH 2013 (June 2011)","DOI":"10.1109\/PTC.2013.6652217"},{"key":"197_CR9","unstructured":"Cellina F, Bettini A, Eva D, Rudel R (2021) Literature review regarding future mobility scenarios. Technical report, SUPSI, Mendrisio. https:\/\/evaproject.eu\/wp-content\/uploads\/2021\/04\/EVA_D31.pdf"},{"key":"197_CR10","doi-asserted-by":"crossref","unstructured":"Chang F, Chen T, Su W, Alsafasfeh Q (2019) Charging Control of an Electric Vehicle Battery Based on Reinforcement Learning. 10th International Renewable Energy Congress, IREC 2019 (March) (2019)","DOI":"10.1109\/IREC.2019.8754518"},{"key":"197_CR11","unstructured":"Chou PW, Maturana D, Scherer S (2017) Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. 34th International Conference on Machine Learning, ICML 2017 2, 1386\u20131396"},{"key":"197_CR12","unstructured":"CIGRE\u2019 Task Force C6.04.02: Benchmark systems for network integration of renewable and distributed energy resources. Technical report, CIGRE International Council on large electric systems (July 2009)"},{"key":"197_CR13","doi-asserted-by":"crossref","unstructured":"Ding T, Zeng Z, Bai J, Qin B, Qin B, Yang Y, Shahidehpour M, Shahidehpour M (2020) Optimal electric vehicle charging strategy with markov decision process and reinforcement learning technique. IEEE Transactions on Industry Applications","DOI":"10.1109\/TIA.2020.2990096"},{"issue":"August","key":"197_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.apenergy.2021.117504","volume":"301","author":"M Dorokhova","year":"2021","unstructured":"Dorokhova M, Martinson Y, Ballif C, Wyrsch N (2021) Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation. Appl Energy 301(August):117504","journal-title":"Appl Energy"},{"key":"197_CR15","doi-asserted-by":"crossref","unstructured":"Dubey A, Santoso S (2015) Electric vehicle charging on residential distribution systems: impacts and mitigations. IEEE Access","DOI":"10.1109\/ACCESS.2015.2476996"},{"key":"197_CR16","doi-asserted-by":"publisher","DOI":"10.5334\/baw.1","volume-title":"Introducing MATsim","author":"A Horni","year":"2016","unstructured":"Horni A, Nagel K, Axhausen KW (2016) Introducing MATsim. Ubiquity Press, London"},{"key":"197_CR17","unstructured":"Kirk R, Zhang A, Grefenstette E, Rockt\u00e4schel T (2021) A survey of generalisation in deep reinforcement learning, 1\u201343. 2111.09794"},{"key":"197_CR18","unstructured":"Matloff N (2008) Introduction to discrete-event simulation and the simpy language. Davis, CA. Dept of Computer Science. University of California at Davis. Retrieved on August 2(2009), 1\u201333"},{"key":"197_CR19","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.tra.2018.12.003","volume":"120","author":"R Mounce","year":"2019","unstructured":"Mounce R, Nelson JD (2019) On the potential for one-way electric vehicle car-sharing in future mobility systems. Transp Res Part A Policy Pract 120:17\u201330","journal-title":"Transp Res Part A Policy Pract"},{"key":"197_CR20","unstructured":"Nespoli L (2019) Model based forecasting for demand response strategies. PhD thesis"},{"key":"197_CR21","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, KilleenT, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A,Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S,Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: An imperativestyle, high-performance deep learning library. In: Wallach, H.,Larochelle, H., Beygelzimer, A., d\u2019 Alch\u00e9-Buc, F., Fox, E.,Garnett, R. (eds.) Advances in Neural Information Processing Systems32, pp. 8024\u20138035. Curran Associates, Inc., http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"197_CR22","doi-asserted-by":"crossref","unstructured":"Peters J, Bagnell JA (2016) Systems, I.A.: Policy Gradient Methods, 1\u20134","DOI":"10.1007\/978-1-4899-7502-7_646-1"},{"key":"197_CR23","doi-asserted-by":"crossref","unstructured":"Petrazzini IGB, Antonelo EA (2022) Proximal policy optimization with continuous bounded action space via the beta distribution, 1\u20138","DOI":"10.1109\/SSCI50451.2021.9660123"},{"key":"197_CR24","doi-asserted-by":"publisher","first-page":"384","DOI":"10.1016\/j.renene.2018.02.126","volume":"125","author":"A Rauf","year":"2018","unstructured":"Rauf A, Salam Z (2018) A rule-based energy management scheme for uninterrupted electric vehicles charging at constant price using photovoltaic-grid system. Renewable Energy 125:384\u2013400","journal-title":"Renewable Energy"},{"issue":"1","key":"197_CR25","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1109\/TPWRS.2011.2158247","volume":"27","author":"P Richardson","year":"2011","unstructured":"Richardson P, Flynn D, Keane A (2011) Optimal charging of electric vehicles in low voltage distribution systems. IEEE Trans Power Syst 27(1):268\u2013279","journal-title":"IEEE Trans Power Syst"},{"key":"197_CR26","unstructured":"Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms, 1\u201312"},{"issue":"11","key":"197_CR27","doi-asserted-by":"publisher","first-page":"4822","DOI":"10.1109\/TITS.2019.2947408","volume":"21","author":"J Shi","year":"2020","unstructured":"Shi J, Gao Y, Wang W, Yu N, Ioannou PA (2020) Operating electric vehicle fleet for ride-hailing services with reinforcement learning. IEEE Trans Intell Transp Syst 21(11):4822\u20134834","journal-title":"IEEE Trans Intell Transp Syst"},{"key":"197_CR28","doi-asserted-by":"crossref","unstructured":"Stein JS (2012) The photovoltaic Performance Modeling Collaborative (PVPMC). Conference Record of the IEEE Photovoltaic Specialists Conference, 3048\u20133052","DOI":"10.1109\/PVSC.2012.6318225"},{"issue":"2","key":"197_CR29","doi-asserted-by":"publisher","first-page":"624","DOI":"10.1109\/TSG.2016.2558585","volume":"9","author":"B Sun","year":"2018","unstructured":"Sun B, Huang Z, Tan X, Tsang DHK (2018) Optimal scheduling for electric vehicle charging with discrete charging levels in distribution grid. IEEE Trans Smart Grid 9(2):624\u2013634","journal-title":"IEEE Trans Smart Grid"},{"key":"197_CR30","volume-title":"Reinforcement learning, Second Edition: an introduction, 2nd edn Adaptive Computation and Machine Learning series","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning, Second Edition: an introduction, 2nd edn Adaptive Computation and Machine Learning series, 2nd edn. MIT Press, Massachusetts","edition":"2"},{"key":"197_CR31","unstructured":"Sutton RS, Mcallester D, Singh S, Mansour Y, Avenue P, Park F (1996) Policy gradient methods for reinforcement learning with function approximation. Adv Neural Inf Proc Syst 12"},{"issue":"6","key":"197_CR32","doi-asserted-by":"publisher","first-page":"6510","DOI":"10.1109\/TPWRS.2018.2829021","volume":"33","author":"L Thurner","year":"2018","unstructured":"Thurner L, Scheidler A, Sch\u00e4fer F, Menke J-H, Dollichon J, Meier F, Meinecke S, Braun M (2018) Pandapower-an open-source python tool for convenient modeling, analysis, and optimization of electric power systems. IEEE Trans Power Syst 33(6):6510\u20136521","journal-title":"IEEE Trans Power Syst"},{"key":"197_CR33","doi-asserted-by":"crossref","unstructured":"Wan Z, Li H, He H, Prokhorov DV (2019) Model-free real-time EV charging scheduling based on deep reinforcement learning. IEEE Transactions on Smart Grid","DOI":"10.1109\/TSG.2018.2879572"},{"key":"197_CR34","unstructured":"Wang Z, Bapst V, Heess N, Mnih V, Munos R, Kavukcuoglu K, de Freitas N (2016) Sample efficient actor-critic with experience replay"},{"issue":"6","key":"197_CR35","doi-asserted-by":"publisher","first-page":"3157","DOI":"10.1109\/TCYB.2020.2975134","volume":"51","author":"X Zhang","year":"2021","unstructured":"Zhang X, Chan KW, Li H, Wang H, Qiu J, Wang G (2021) Deep-learning-based probabilistic forecasting of electric vehicle charging load with a novel queuing model. IEEE Trans Cybern 51(6):3157\u20133170","journal-title":"IEEE Trans Cybern"},{"issue":"14","key":"197_CR36","first-page":"1","volume":"12","author":"J Zhu","year":"2019","unstructured":"Zhu J, Yang Z, Mourshed M, Guo Y, Zhou Y, Chang Y, Wei Y, Feng S (2019) Electric vehicle charging load forecasting: a comparative study of deep learning approaches. Energies 12(14):1\u201319","journal-title":"Energies"}],"container-title":["Energy Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s42162-022-00197-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s42162-022-00197-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s42162-022-00197-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T14:21:40Z","timestamp":1662646900000},"score":1,"resource":{"primary":{"URL":"https:\/\/energyinformatics.springeropen.com\/articles\/10.1186\/s42162-022-00197-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,7]]},"references-count":36,"journal-issue":{"issue":"S1","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["197"],"URL":"https:\/\/doi.org\/10.1186\/s42162-022-00197-5","relation":{},"ISSN":["2520-8942"],"issn-type":[{"value":"2520-8942","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,7]]},"assertion":[{"value":"7 September 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"28"}}