{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T16:35:32Z","timestamp":1780763732239,"version":"3.54.1"},"reference-count":47,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,10,6]],"date-time":"2021-10-06T00:00:00Z","timestamp":1633478400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62173032"],"award-info":[{"award-number":["62173032"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Battery energy storage technology is an important part of the industrial parks to ensure the stable power supply, and its rough charging and discharging mode is difficult to meet the application requirements of energy saving, emission reduction, cost reduction, and efficiency increase. As a classic method of deep reinforcement learning, the deep Q-network is widely used to solve the problem of user-side battery energy storage charging and discharging. In some scenarios, its performance has reached the level of human expert. However, the updating of storage priority in experience memory often lags behind updating of Q-network parameters. In response to the need for lean management of battery charging and discharging, this paper proposes an improved deep Q-network to update the priority of sequence samples and the training performance of deep neural network, which reduces the cost of charging and discharging action and energy consumption in the park. The proposed method considers factors such as real-time electricity price, battery status, and time. The energy consumption state, charging and discharging behavior, reward function, and neural network structure are designed to meet the flexible scheduling of charging and discharging strategies, and can finally realize the optimization of battery energy storage benefits. The proposed method can solve the problem of priority update lag, and improve the utilization efficiency and learning performance of the experience pool samples. The paper selects electricity price data from the United States and some regions of China for simulation experiments. Experimental results show that compared with the traditional algorithm, the proposed approach can achieve better performance in both electricity price systems, thereby greatly reducing the cost of battery energy storage and providing a stronger guarantee for the safe and stable operation of battery energy storage systems in industrial parks.<\/jats:p>","DOI":"10.3390\/e23101311","type":"journal-article","created":{"date-parts":[[2021,10,7]],"date-time":"2021-10-07T09:24:04Z","timestamp":1633598644000},"page":"1311","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Improved Deep Q-Network for User-Side Battery Energy Storage Charging and Discharging Strategy in Industrial Parks"],"prefix":"10.3390","volume":"23","author":[{"given":"Shuai","family":"Chen","sequence":"first","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Key Laboratory of Knowledge Automation for Industrial Processes of Ministry of Education, School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chengpeng","family":"Jiang","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Key Laboratory of Knowledge Automation for Industrial Processes of Ministry of Education, School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinglin","family":"Li","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Key Laboratory of Knowledge Automation for Industrial Processes of Ministry of Education, School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinwei","family":"Xiang","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Key Laboratory of Knowledge Automation for Industrial Processes of Ministry of Education, School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2270-7889","authenticated-orcid":false,"given":"Wendong","family":"Xiao","sequence":"additional","affiliation":[{"name":"School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Key Laboratory of Knowledge Automation for Industrial Processes of Ministry of Education, School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"778","DOI":"10.1016\/j.eng.2019.11.015","article-title":"A grid as smart as the Internet","volume":"6","author":"Liu","year":"2020","journal-title":"Engineering"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1016\/j.gloei.2020.05.007","article-title":"Engineering practices for the integration of large-scale renewable energy VSC-HVDC systems","volume":"3","author":"Li","year":"2020","journal-title":"Glob. Energy Interconnect."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Li, J., Chen, S., Jiang, C., Liu, F., and Xiao, W. (2020, January 27\u201329). Adaptive dynamic programming approach for micro-grid optimal energy transmission scheduling. Proceedings of the 39th Chinese Control Conference, Shenyang, China.","DOI":"10.23919\/CCC50068.2020.9189579"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gao, Y., Li, J., and Hong, M. (2021). Machine learning based optimization model for energy management of energy storage system for large industrial park. Processes, 9.","DOI":"10.3390\/pr9050825"},{"key":"ref_5","first-page":"116","article-title":"Joint planning of battery energy storage and demand response for industrial park participating in peak shaving","volume":"43","author":"Hu","year":"2019","journal-title":"Autom. Electr. Power Syst."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Qin, Y., Lin, H., Tan, Z., Yan, Q., Li, L., Yang, S., De, G., and Ju, L. (2019). A dispatching optimization model for park power supply systems considering power-to-gas and peak regulation compensation. Processes, 7.","DOI":"10.3390\/pr7110813"},{"key":"ref_7","first-page":"62","article-title":"Operation control strategy for energy storage station after considering battery life in commercial park","volume":"46","author":"Li","year":"2020","journal-title":"High Volt. Eng."},{"key":"ref_8","first-page":"59","article-title":"Optimized allocation of microgrid energy storage capacity considering photovoltaic correction","volume":"49","author":"Chen","year":"2021","journal-title":"Power Syst. Prot. Control."},{"key":"ref_9","first-page":"44","article-title":"Control strategies of battery energy storage system participating in peak load regulation of power grid","volume":"50","author":"Zhou","year":"2021","journal-title":"Therm. Power Gener."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2882","DOI":"10.1109\/TPWRS.2017.2749512","article-title":"Using battery storage for peak shaving and frequency regulation: Joint optimization for superlinear gains","volume":"33","author":"Shi","year":"2018","journal-title":"IEEE Trans. Power Syst."},{"key":"ref_11","first-page":"44","article-title":"Optimal combined peak-shaving scheme using energy storage for auxiliary considering both technology and economy","volume":"41","author":"Li","year":"2017","journal-title":"Autom. Electr. Power Syst."},{"key":"ref_12","first-page":"152","article-title":"Optimal dispatch of power system considering uncertainty of demand response based on extended sequence operation","volume":"42","author":"Xu","year":"2018","journal-title":"Autom. Electr. Power Syst."},{"key":"ref_13","first-page":"1901","article-title":"Study on the optimal utilization of integrated energy system emergency reserve based on risk quantification and demand side response","volume":"36","author":"Liu","year":"2021","journal-title":"Trans. China Electrotech. Soc."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"3346","DOI":"10.1109\/TPDS.2014.2372781","article-title":"Distributed demand side management with energy storage in smart grid","volume":"26","author":"Nguyen","year":"2015","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"ref_15","first-page":"27","article-title":"SOC management strategy of storage system and its impact assessment","volume":"39","author":"Li","year":"2015","journal-title":"Autom. Electr. Power Syst."},{"key":"ref_16","unstructured":"Hoog, J.D., Abdulla, K., Kolluri, R.R., and Karki, P. (2018, January 12\u201315). Scheduling fast local rule-based controllers for optimal operation of energy storage. Proceedings of the 9th International Conference on Future Energy Systems, New York, NY, USA."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1937","DOI":"10.1109\/TPDS.2014.2329491","article-title":"Joint workload and battery scheduling with heterogeneous service delay guarantees for data center energy cost minimization","volume":"26","author":"Yu","year":"2015","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"ref_18","first-page":"703","article-title":"Energy efficiency optimization based on storage scheduling and multi-source power supplying of data center in energy Internet","volume":"54","author":"Sun","year":"2017","journal-title":"J. Comput. Res. Dev."},{"key":"ref_19","first-page":"150","article-title":"Energy storage control strategy of wind-photovoltaic-storage hybrid system","volume":"49","author":"Niu","year":"2020","journal-title":"Therm. Power Gener."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1016\/j.apenergy.2017.05.176","article-title":"Study on battery pack consistency evolutions and equilibrium diagnosis for serial-connected lithium-ion batteries","volume":"207","author":"Zhang","year":"2017","journal-title":"Appl. Energy"},{"key":"ref_21","unstructured":"Guan, C., Wang, Y., Lin, X., Nazarian, S., and Pedram, M. (2015, January 9\u201312). Reinforcement learning-based control of residential energy storage systems for electric bill minimization. Proceedings of the 12th Annual IEEE Consumer Communications and Networking Conference, Las Vegas, NV, USA."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3126495","article-title":"Adaptive power management in solar energy harvesting sensor node using reinforcement learning","volume":"16","author":"Shresthamali","year":"2017","journal-title":"ACM Trans. Embed. Comput. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lee, J., Issimova, Z., and Lim, H. (2018, January 12\u201315). Reinforcement learning based charging station management for electric vehicles. Proceedings of the 9th International Conference on Future Energy Systems, New York, NY, USA.","DOI":"10.1145\/3208903.3212057"},{"key":"ref_24","first-page":"404","article-title":"State of charge of Li-ion batteries based on reinforcement learning","volume":"51","author":"Kim","year":"2018","journal-title":"IFAC Pap."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1016\/j.neucom.2019.08.106","article-title":"EV charging bidding by multi-DQN reinforcement learning in electricity auction market","volume":"397","author":"Zhang","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_26","first-page":"358","article-title":"Data-driven decision-making strategies for electricity retailers: A deep reinforcement learning approach","volume":"7","author":"Liu","year":"2021","journal-title":"CSEE J. Power Energy Syst."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hasselt, H.V., Guez, A., and Silver, D. (2016, January 12\u201317). Deep reinforcement learning with double q-learning. Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_28","unstructured":"Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016, January 2\u20134). Prioritized experience replay. Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico."},{"key":"ref_29","unstructured":"Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., and de Freitas, N. (2016, January 19\u201324). Dueling network architectures for deep reinforcement learning. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Sidorov, D., Panasetsky, D., Tomin, N., Karamov, D., Zhukov, A., Muftahov, I., Dreglea, A., Liu, F., and Li, Y. (2020). Toward zero-emission hybrid AC\/DC power systems with renewable energy sources and storages: A case study from lake Baikal region. Energies, 13.","DOI":"10.3390\/en13051226"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"3451","DOI":"10.1109\/TII.2019.2932453","article-title":"A dynamic analysis of energy storage with renewable and diesel generation using Volterra equations","volume":"16","author":"Sidorov","year":"2020","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2100199","DOI":"10.1002\/ente.202100199","article-title":"Optimized integration of hybrid renewable sources with long-life battery energy storage in microgrids for peak power shaving and demand side management under different tariff scenario","volume":"9","author":"Bhattacharjee","year":"2021","journal-title":"Energy Technol."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Bhattacharjee, A., Mohanty, R.K., and Ghosh, A. (2020). Design of an optimized thermal management system for Li-ion batteries under different discharging conditions. Energies, 13.","DOI":"10.3390\/en13215695"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Settele, F., Holzapfel, F., and Knoll, A. (2020). The impact of Peukert-effect on optimal control of a battery-electrically driven airplane. Aerospace, 7.","DOI":"10.3390\/aerospace7020013"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1016\/j.apenergy.2008.11.021","article-title":"Enhanced coulomb counting method for estimating state-of-charge and state-of-health of lithium-ion batteries","volume":"86","author":"Ng","year":"2009","journal-title":"Appl. Energy"},{"key":"ref_36","first-page":"206","article-title":"Sequence to sequence multi-agent reinforcement learning algorithm","volume":"34","author":"Shi","year":"2021","journal-title":"Pattern Recognit. Artif. Intell."},{"key":"ref_37","first-page":"1696","article-title":"Reinforcement learning driven by self-optimizing operation for distributed electrical storage system","volume":"44","author":"Shao","year":"2020","journal-title":"Power Syst. Technol."},{"key":"ref_38","first-page":"1914","article-title":"A coordinated control method for hybrid energy storage system in microgrid based on deep reinforcement learning","volume":"43","author":"Zhang","year":"2019","journal-title":"Power Syst. Technol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"542","DOI":"10.1109\/TITS.2020.3002271","article-title":"Effective charging planning based on deep reinforcement learning for electric vehicles","volume":"22","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_40","first-page":"22","article-title":"A deep neural network coordination model for electric heating and cooling loads based on IoT data","volume":"6","author":"Jin","year":"2020","journal-title":"CSEE J. Power Energy Syst."},{"key":"ref_41","first-page":"1140","article-title":"Multi-agent deep reinforcement learning algorithm based on partitioned buffer replay and multiple process interaction","volume":"44","author":"Chai","year":"2021","journal-title":"Chin. J. Comput."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Yi, C., and Qi, M. (2020, January 28\u201329). Research on virtual path planning based on improved DQN. Proceedings of the IEEE International Conference on Real-time Computing and Robotics, Asahikawa, Japan.","DOI":"10.1109\/RCAR49640.2020.9303290"},{"key":"ref_43","first-page":"6404819","article-title":"Mechanical parameter identification of hydraulic engineering with the improved deep Q-network algorithm","volume":"2020","author":"Wei","year":"2020","journal-title":"Math. Probl. Eng."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.ins.2020.03.105","article-title":"Deep reinforcement learning for pedestrian collision avoidance and human-machine cooperative driving","volume":"532","author":"Li","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"502","DOI":"10.4208\/jcm.1901-m2018-0160","article-title":"ReLU deep neural networks and linear finite elements","volume":"38","author":"He","year":"2020","journal-title":"J. Comput. Math."},{"key":"ref_46","unstructured":"(2021, September 30). Baltimore Gas and Electric Company. Available online: https:\/\/supplier.bge.com\/LoadProfilesEnergySettlement\/historicalloaddata.html."},{"key":"ref_47","unstructured":"(2021, September 30). Beijing Sales Electricity Price Table, Available online: http:\/\/www.beijing.gov.cn\/fuwu\/bmfw\/jmsh\/jmshshjf\/shjfd\/dj\/t1492381.html."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/10\/1311\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:09:53Z","timestamp":1760166593000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/10\/1311"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,6]]},"references-count":47,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["e23101311"],"URL":"https:\/\/doi.org\/10.3390\/e23101311","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,6]]}}}