{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:02:33Z","timestamp":1760144553141,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2024,4,30]],"date-time":"2024-04-30T00:00:00Z","timestamp":1714435200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["62173137","52172403","62303178","23A0426","22B0577"],"award-info":[{"award-number":["62173137","52172403","62303178","23A0426","22B0577"]}]},{"name":"Project of the Hunan Provincial Department of Education, China","award":["62173137","52172403","62303178","23A0426","22B0577"],"award-info":[{"award-number":["62173137","52172403","62303178","23A0426","22B0577"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Cyclic air braking is a key element for ensuring safe train operation when running on a long and steep downhill railway section. In reality, the cyclic braking performance of a train is affected by its operating environment, speed and air-refilling time. Existing optimization algorithms have the problem of low learning efficiency. To solve this problem, an intelligent control method based on the deep Q-network (DQN) algorithm for heavy-haul trains running on long and steep downhill railway sections is proposed. Firstly, the environment of heavy-haul train operation is designed by considering the line characteristics, speed limits and constraints of the train pipe\u2019s air-refilling time. Secondly, the control process of heavy-haul trains running on long and steep downhill sections is described as the reinforcement learning (RL) of a Markov decision process. By designing the critical elements of RL, a cyclic braking strategy for heavy-haul trains is established based on the reinforcement learning algorithm. Thirdly, the deep neural network and Q-learning are combined to design a neural network for approximating the action value function so that the algorithm can achieve the optimal action value function faster. Finally, simulation experiments are conducted on the actual track data pertaining to the Shuozhou\u2013Huanghua line in China to compare the performance of the Q-learning algorithm against the DQN algorithm. Our findings revealed that the DQN-based intelligent control strategy decreased the air braking distance by 2.1% and enhanced the overall average speed by more than 7%. These experiments unequivocally demonstrate the efficacy and superiority of the DQN-based intelligent control strategy.<\/jats:p>","DOI":"10.3390\/a17050190","type":"journal-article","created":{"date-parts":[[2024,4,30]],"date-time":"2024-04-30T08:14:31Z","timestamp":1714464871000},"page":"190","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains"],"prefix":"10.3390","volume":"17","author":[{"given":"Changfan","family":"Zhang","sequence":"first","affiliation":[{"name":"College of Electrical and Information Engineering, Hunan University of Technology, Zhuzhou 412007, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5848-5840","authenticated-orcid":false,"given":"Shuo","family":"Zhou","sequence":"additional","affiliation":[{"name":"College of Electrical and Information Engineering, Hunan University of Technology, Zhuzhou 412007, China"}]},{"given":"Jing","family":"He","sequence":"additional","affiliation":[{"name":"College of Electrical and Information Engineering, Hunan University of Technology, Zhuzhou 412007, China"}]},{"given":"Lin","family":"Jia","sequence":"additional","affiliation":[{"name":"College of Electrical and Information Engineering, Hunan University of Technology, Zhuzhou 412007, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,30]]},"reference":[{"key":"ref_1","unstructured":"Zhang, Z. (2017). Optimization Analysis of Smooth Operation for Ten-Thousand Ton Trains of Shuohuang Railway, Southwest Publishing House."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lu, Q., He, B., Wu, M., Zhang, Z., Luo, J., Zhang, Y., He, R., and Wang, K. (2018). Establishment and analysis of energy consumption model of heavy-haul train on large long slope. Energies, 11.","DOI":"10.3390\/en11040965"},{"key":"ref_3","first-page":"28","article-title":"Analysis of Braking Methods for Express Freight Train on Long Ramp","volume":"2","author":"Dong","year":"2022","journal-title":"Mod. Mach."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kuefler, A., Morton, J., Wheeler, T., and Kochenderfer, M. (2017, January 11\u201314). Imitating Driver Behavior with Generative Adversarial Networks. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.","DOI":"10.1109\/IVS.2017.7995721"},{"key":"ref_5","first-page":"799","article-title":"Expert supervised SAC reinforcement learning for optimizing the operation of heavy-duty trains","volume":"39","author":"Yang","year":"2022","journal-title":"Control Theory Appl."},{"key":"ref_6","first-page":"454","article-title":"An optimal strategy for the control of a train","volume":"31","author":"Howlett","year":"1990","journal-title":"Anziam J."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Huang, Y., Su, S., and Liu, W. (2020, January 20\u201323). optimization on the Driving Curve of Heavy Haul Trains Based on Artificial Bee Colony Algorithm. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.","DOI":"10.1109\/ITSC45102.2020.9294214"},{"key":"ref_8","unstructured":"Du, L. (2021). Model Based Security Reinforcement Learning. [Master\u2019s Thesis, Harbin Institute of Technology]."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.trc.2013.01.007","article-title":"Optimal trajectory planning for trains\u2014A pseudo spectral method and a mixed integer linear programming approach","volume":"29","author":"Wang","year":"2013","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_10","first-page":"111","article-title":"A Study on Energy-Saving Optimization for High Speed Railways Based on Q-learning Algorithm","volume":"41","author":"Zhang","year":"2019","journal-title":"Railw. Transp. Econ."},{"key":"ref_11","first-page":"69","article-title":"A Policy-Based Reinforcement Learning Algorithm for Intelligent Train Control","volume":"42","author":"Zhang","year":"2020","journal-title":"J. China Railw. Soc."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhang, W., Sun, X., Liu, Z., and Yang, L. (2023, January 24\u201328). Research on Energy-Saving Speed Curve of Heavy Haul Train Based on Reinforcement Learning. Proceedings of the 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), Bilbao, Bizkaia.","DOI":"10.1109\/ITSC57777.2023.10422218"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"37351","DOI":"10.1007\/s11042-023-15051-3","article-title":"A comprehensive study on reinforcement learning application for train speed profile optimization","volume":"82","author":"Sandidzadeh","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"3491","DOI":"10.1016\/j.ifacol.2023.10.1503","article-title":"A Deep Reinforcement Learning Approach for Optimal Scheduling of Heavy-haul Railway","volume":"56","author":"Wu","year":"2023","journal-title":"IFAC-PapersOnLine"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"105605","DOI":"10.1016\/j.conengprac.2023.105605","article-title":"Reinforcement learning method for the multi-objective speed trajectory optimization of a freight train","volume":"138","author":"Lin","year":"2023","journal-title":"Control Eng. Pract."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"105173","DOI":"10.1016\/j.knosys.2019.105173","article-title":"Reinforcement learning approach for optimal control of multiple electric locomotives in a heavy-haul freight train: A Double-Switch-Q-network architecture","volume":"190","author":"Tang","year":"2020","journal-title":"Knowl.-Based Syst."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, M.Y., Kou, B.Q., and Zhao, X.K. (2018, January 7\u201310). Analysis of Energy Consumption Characteristics Based on Simulation and Traction Calculation Model for the CRH Electric Motor Train Units. Proceedings of the 2018 21st International Conference on Electrical Machines and Systems (ICEMS), Jeju, Republic of Korea.","DOI":"10.23919\/ICEMS.2018.8549182"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yu, H., Huang, Y., and Wang, M. (2018, January 4\u20137). Research on operating strategy based on particle swarm optimization for heavy haul train on long down-slope. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569904"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1687814018791016","DOI":"10.1177\/1687814018791016","article-title":"Research on the driving strategy of heavy-haul train based on improved genetic algorithm","volume":"10","author":"Huang","year":"2018","journal-title":"Adv. Mech. Eng."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1177\/0954409716671546","article-title":"Driving strategy optimization for trains in subway systems","volume":"232","author":"Su","year":"2018","journal-title":"Proc. Inst. Mech. Eng. Part F J. Rail Rapid Transit"},{"key":"ref_21","first-page":"663","article-title":"Research on Energy-saving Operation of High-speed Trains Based on Improved Genetic Algorithm","volume":"26","author":"Niu","year":"2022","journal-title":"J. Appl. Sci. Eng."},{"key":"ref_22","unstructured":"Zhang, Z. (2013). Train Traction Calculation, China Railway Press."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4252","DOI":"10.1109\/TITS.2019.2939358","article-title":"An energy-efficient train operation approach by integrating the metro timetabling and eco-driving","volume":"21","author":"Su","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_24","unstructured":"Ma, H. (2019). Research on Optimization Control of Long Downhill Braking Process for Heavy Haul Trains. [Master\u2019s Thesis, East China Jiaotong University]."},{"key":"ref_25","unstructured":"Wu, J. (2013). Train Traction Calculation, Southwest Jiaotong University Press."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1093\/tse\/tdaa015","article-title":"Study on a Segmented Electro-Pneumatic Braking System for Heavy-Haul Trains","volume":"2","author":"Wei","year":"2020","journal-title":"Transp. Saf. Environ."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1109\/MITS.2019.2907681","article-title":"Design of Running Grades for Energy-Efficient Train Regulation: A Case Study for Beijing Yizhuang Line","volume":"13","author":"Su","year":"2019","journal-title":"IEEE Intell. Transp. Syst. Mag."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/5\/190\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:36:59Z","timestamp":1760107019000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/5\/190"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,30]]},"references-count":27,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["a17050190"],"URL":"https:\/\/doi.org\/10.3390\/a17050190","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2024,4,30]]}}}