{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T19:19:32Z","timestamp":1786216772987,"version":"3.56.0"},"reference-count":44,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,20]],"date-time":"2023-01-20T00:00:00Z","timestamp":1674172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2022YFC3203800"],"award-info":[{"award-number":["2022YFC3203800"]}]},{"name":"National Key Research and Development Program of China","award":["51978203"],"award-info":[{"award-number":["51978203"]}]},{"name":"National Key Research and Development Program of China","award":["CE602022000203"],"award-info":[{"award-number":["CE602022000203"]}]},{"name":"National Natural Science Foundation of China","award":["2022YFC3203800"],"award-info":[{"award-number":["2022YFC3203800"]}]},{"name":"National Natural Science Foundation of China","award":["51978203"],"award-info":[{"award-number":["51978203"]}]},{"name":"National Natural Science Foundation of China","award":["CE602022000203"],"award-info":[{"award-number":["CE602022000203"]}]},{"name":"Unveiling Scientific Research Program","award":["2022YFC3203800"],"award-info":[{"award-number":["2022YFC3203800"]}]},{"name":"Unveiling Scientific Research Program","award":["51978203"],"award-info":[{"award-number":["51978203"]}]},{"name":"Unveiling Scientific Research Program","award":["CE602022000203"],"award-info":[{"award-number":["CE602022000203"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>Effective ways to optimise real-time pump scheduling to maximise energy efficiency are being sought to meet the challenges in the energy market. However, the considerable number of evaluations of popular optimisation methods based on metaheuristics cause significant delays for real-time pump scheduling, and the simplification of traditional deterministic methods may introduce bias towards the optimal solutions. To address these limitations, an exploration-enhanced deep reinforcement learning (DRL) framework is proposed to address real-time pump scheduling problems in water distribution systems. The experimental results indicate that E-PPO can learn suboptimal scheduling policies for various demand distributions and can control the application time to 0.42 s by transferring the online computation-intensive optimisation task offline. Furthermore, a form of penalty of the tank level was found that can reduce energy costs by up to 11.14% without sacrificing the water level in the long term. Following the DRL framework, the proposed method makes it possible to schedule pumps in a more agile way as a timely response to changing water demand while still controlling the energy cost and level of tanks.<\/jats:p>","DOI":"10.3390\/systems11020056","type":"journal-article","created":{"date-parts":[[2023,1,23]],"date-time":"2023-01-23T03:48:08Z","timestamp":1674445688000},"page":"56","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["Real-Time Scheduling of Pumps in Water Distribution Systems Based on Exploration-Enhanced Deep Reinforcement Learning"],"prefix":"10.3390","volume":"11","author":[{"given":"Shiyuan","family":"Hu","sequence":"first","affiliation":[{"name":"School of Environment, Harbin Institute of Technology, Harbin 150090, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6662-0187","authenticated-orcid":false,"given":"Jinliang","family":"Gao","sequence":"additional","affiliation":[{"name":"School of Environment, Harbin Institute of Technology, Harbin 150090, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dan","family":"Zhong","sequence":"additional","affiliation":[{"name":"School of Environment, Harbin Institute of Technology, Harbin 150090, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Wu","sequence":"additional","affiliation":[{"name":"Guangdong Yuehai Water Investment Co., Ltd., Shenzhen 518021, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luming","family":"Liu","sequence":"additional","affiliation":[{"name":"National Engineering Research Center of Urban Water Resources Co., Ltd., Harbin Institute of Technology, Harbin 150090, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1061\/(ASCE)0733-9496(2004)130:2(160)","article-title":"Operational optimization of water distribution systems using a hybrid genetic algorithm","volume":"130","author":"Savic","year":"2004","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1016\/j.envsoft.2017.02.009","article-title":"Lost in optimisation of water distribution systems? A literature review of system operation","volume":"93","author":"Sultanova","year":"2017","journal-title":"Environ. Modell. Softw."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1016\/j.proeng.2015.08.936","article-title":"Pumping Pattern Optimization in Order to Reduce WDS Operation Costs","volume":"119","author":"Saldarriaga","year":"2015","journal-title":"Procedia Eng."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Cimorelli, L., Covelli, C., Molino, B., and Pianese, D. (2020). Optimal Regulation of Pumping Station in Water Distribution Networks Using Constant and Variable Speed Pumps: A Technical and Economical Comparison. Energies, 13.","DOI":"10.3390\/en13102530"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1016\/j.egypro.2019.01.002","article-title":"Impacts of extreme climate conditions due to climate change on the energy system design and operation","volume":"159","author":"Perera","year":"2019","journal-title":"Energy Procedia"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"113403","DOI":"10.1016\/j.jenvman.2021.113403","article-title":"Energy market uncertainty and the impact on the crude oil prices","volume":"298","author":"Xu","year":"2021","journal-title":"J. Environ. Manag."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"106657","DOI":"10.1016\/j.resconrec.2022.106657","article-title":"Influence of Russia-Ukraine War on the Global Energy and Food Security","volume":"188","author":"Zhou","year":"2023","journal-title":"Resour. Conserv. Recycl."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"101360","DOI":"10.1016\/j.ribaf.2020.101360","article-title":"Does COVID-19 open a Pandora\u2019s box of changing the connectedness in energy commodities?","volume":"56","author":"Lin","year":"2021","journal-title":"Res. Int. Bus. Financ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"105498","DOI":"10.1016\/j.eneco.2021.105498","article-title":"The risk spillover effect of the COVID-19 pandemic on energy sector: Evidence from China","volume":"102","author":"Si","year":"2021","journal-title":"Energy Econ."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1016\/j.apenergy.2016.06.114","article-title":"Energy consumption for water use cycles in different countries: A review","volume":"178","author":"Wakeel","year":"2016","journal-title":"Appl. Energy"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1061\/(ASCE)0733-9496(1992)118:4(406)","article-title":"Optimal Pump Scheduling in Water-Supply Networks","volume":"118","author":"Jowitt","year":"1992","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1007\/BF02207638","article-title":"Optimized pump scheduling in water distribution systems","volume":"83","author":"Yu","year":"1994","journal-title":"J. Optimiz. Theory Appl."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1551","DOI":"10.1061\/(ASCE)0733-9429(1991)117:11(1551)","article-title":"Methodology for Optimal Operation of Pumping Stations in Water Distribution Systems","volume":"117","author":"Brion","year":"1991","journal-title":"J. Hydraul. Eng."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Maskit, M., and Ostfeld, A. (2021). Multi-Objective Operation-Leakage Optimization and Calibration of Water Distribution Systems. Water, 13.","DOI":"10.3390\/w13111606"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1061\/(ASCE)0733-9496(1994)120:1(17)","article-title":"Optimal Pump Operations Considering Pump Switches","volume":"120","author":"Lansey","year":"1994","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.1016\/0005-1098(93)90048-X","article-title":"Applied mathematics in water supply network management","volume":"29","author":"Carpentier","year":"1993","journal-title":"Automatica"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1016\/S1474-6670(17)51596-7","article-title":"Operational Control of Integrated Quality and Quantity in Water Systems","volume":"28","author":"Brdys","year":"1995","journal-title":"IFAC Proc. Vol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"155","DOI":"10.2166\/wst.2002.0228","article-title":"Optimal operation of a potable water distribution network","volume":"46","author":"Biscos","year":"2002","journal-title":"Water Sci. Technol. J. Int. Assoc. Water Pollut. Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1061\/(ASCE)WR.1943-5452.0000239","article-title":"Fast Hybrid Optimization Method for Effective Pump Scheduling","volume":"139","author":"Giacomello","year":"2013","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Geem, Z.W. (2005, January 27\u201329). Harmony Search in Water Pump Switching Problem. Proceedings of the International Conference on Natural Computation, Changsha, China.","DOI":"10.1007\/11539902_92"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"4014094","DOI":"10.1061\/(ASCE)WR.1943-5452.0000493","article-title":"Optimization Research of Parallel Pump System for Improving Energy Efficiency","volume":"141","author":"Wu","year":"2015","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Mackle, G., Savic, G.A., and Walters, G.A. (1995, January 12\u201314). Application of genetic algorithms to pump scheduling for water supply. Proceedings of the First International Conference on Genetic Algorithms in Engineering Systems: Innovations and Applications, Sheffield, UK.","DOI":"10.1049\/cp:19951082"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zhu, J., Wang, J., and Li, X. (2016, January 27\u201329). Optimal scheduling of water-supply pump stations based on improved adaptive genetic algorithm. Proceedings of the 2016 35th Chinese Control Conference (CCC), Chengdu, China.","DOI":"10.1109\/ChiCC.2016.7553775"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1061\/(ASCE)0887-3801(1987)1:2(128)","article-title":"Genetic Algorithms in Pipeline Optimization","volume":"1","author":"Goldberg","year":"1987","journal-title":"J. Comput. Civ. Eng."},{"key":"ref_25","unstructured":"Wegley, C., Eusuff, M., and Lansey, K. (August, January 30). Determining Pump Operations using Particle Swarm Optimization. In Proceeding of the Joint Conference on Water Resource Engineering and Water Resources Planning and Management, Minneapolis, MN, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Al-Ani, D., and Habibi, S. (May, January 29). Optimal pump operation for water distribution systems using a new multi-agent Particle Swarm Optimization technique with EPANET. Proceedings of the 2012 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), Montreal, QC, Canada.","DOI":"10.1109\/CCECE.2012.6335031"},{"key":"ref_27","first-page":"20170879","article-title":"A systematic mixed-integer differential evolution approach for water network operational optimization","volume":"474","author":"Zhao","year":"2018","journal-title":"Proc. R. Soc. A Math. Phys. Eng. Sci."},{"key":"ref_28","unstructured":"Li, D., Chen, D., Jin, B., Shi, L., Goh, J., and Ng, S. (2019). Artificial Neural Networks and Machine Learning\u2014ICANN 2019: Text and Time Series, Springer."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"109020","DOI":"10.1016\/j.measurement.2021.109020","article-title":"A steerable pyramid autoencoder based framework for anomaly frame detection of water pipeline CCTV inspection","volume":"174","author":"Jiao","year":"2021","journal-title":"Measurement"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"147647","DOI":"10.1109\/ACCESS.2020.3015655","article-title":"A Two-Layer Water Demand Prediction System in Urban Areas Based on Micro-Services and LSTM Neural Networks","volume":"8","author":"Nasser","year":"2020","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1007\/s10661-017-6030-3","article-title":"Water demand forecasting: Review of soft computing methods","volume":"189","author":"Ghalehkhondabi","year":"2017","journal-title":"Environ. Monit. Assess."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"255","DOI":"10.2166\/wst.2002.0687","article-title":"Application of short-term water demand prediction model to Seoul","volume":"46","author":"Joo","year":"2002","journal-title":"Water Sci. Technol."},{"key":"ref_33","unstructured":"Geem, Z.W., Tseng, C., Kim, J., and Bae, C. (2007). Pipelines 2007, American Society of Civil Engineers."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"14757","DOI":"10.1007\/s00500-021-06177-3","article-title":"Zone scheduling optimization of pumps in water distribution networks with deep reinforcement learning and knowledge-assisted learning","volume":"25","author":"Xu","year":"2021","journal-title":"Soft Comput."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1061\/(ASCE)0733-9496(2003)129:6(458)","article-title":"Neural Networks and Reinforcement Learning in Control of Water Systems","volume":"129","author":"Bhattacharya","year":"2003","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"118973","DOI":"10.1016\/j.watres.2022.118973","article-title":"The role of deep learning in urban water management: A critical review","volume":"223","author":"Fu","year":"2022","journal-title":"Water Res."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1061\/(ASCE)0733-9496(2005)131:3(172)","article-title":"Water Distribution System Optimization Using Metamodels","volume":"131","author":"Broad","year":"2005","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"4020079","DOI":"10.1061\/(ASCE)WR.1943-5452.0001287","article-title":"Deep Reinforcement Learning for Real-Time Optimization of Pumps in Water Distribution Systems","volume":"146","year":"2020","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_40","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jang, S., and Kim, H. (2022). Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning. Sensors, 22.","DOI":"10.2139\/ssrn.4047895"},{"key":"ref_42","unstructured":"Varno, F., Soleimani, B.H., Saghayi, M., Di Jorio, L., and Matwin, S. (2019). Efficient Neural Task Adaptation by Maximum Entropy Initialization. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"04022037","DOI":"10.1061\/(ASCE)WR.1943-5452.0001559","article-title":"Hydraulic Model Database for Applied Water Distribution Systems Research","volume":"148","author":"Ormsbee","year":"2022","journal-title":"J. Water Res. Plan. Man."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"873","DOI":"10.1016\/j.mcm.2012.09.015","article-title":"An algorithm for minimization of pumping costs in water distribution systems using a novel approach to pump scheduling","volume":"57","author":"Bagirov","year":"2013","journal-title":"Math. Comput. Model."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/11\/2\/56\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:12:03Z","timestamp":1760119923000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/11\/2\/56"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,20]]},"references-count":44,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["systems11020056"],"URL":"https:\/\/doi.org\/10.3390\/systems11020056","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,20]]}}}