{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T22:49:15Z","timestamp":1784674155351,"version":"3.55.0"},"reference-count":26,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,3,25]],"date-time":"2022-03-25T00:00:00Z","timestamp":1648166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100018537","name":"National Science and Technology Major Project","doi-asserted-by":"publisher","award":["2017-V-0015-0067"],"award-info":[{"award-number":["2017-V-0015-0067"]}],"id":[{"id":"10.13039\/501100018537","id-type":"DOI","asserted-by":"publisher"}]},{"name":"AECC Sichuan Gas Turbine Establishment Stable Support Project","award":["GJCZ-0011-19"],"award-info":[{"award-number":["GJCZ-0011-19"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>To solve the problem of transient control design with uncertainties and degradation in the life cycle, a design method for a turbofan engine\u2019s transient controller based on reinforcement learning is proposed. The method adopts an actor\u2013critic framework and deep deterministic policy gradient (DDPG) algorithm with the ability to train an agent with continuous action policy for the continuous and violent turbofan engine state change. Combined with a symmetrical acceleration and deceleration transient control plan, a reward function with the aim of servo tracking is proposed. Simulations under different conditions were carried out with a controller designed via the proposed method. The simulation results show that during the acceleration process of the engine from idle to an intermediate state, the controlled variables have no overshoot, and the settling time does not exceed 3.8 s. During the deceleration process of the engine from an intermediate state to idle, the corrected speed of high-pressure rotor has no overshoot, the corrected-speed overshoot of the low-pressure rotor does not exceed 1.5%, and the settling time does not exceed 3.3 s. A system with the designed transient controller can maintain the performance when uncertainties and degradation are considered.<\/jats:p>","DOI":"10.3390\/sym14040684","type":"journal-article","created":{"date-parts":[[2022,3,27]],"date-time":"2022-03-27T21:31:25Z","timestamp":1648416685000},"page":"684","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Transient Controller Design Based on Reinforcement Learning for a Turbofan Engine with Actuator Dynamics"],"prefix":"10.3390","volume":"14","author":[{"given":"Keqiang","family":"Miao","sequence":"first","affiliation":[{"name":"School of Energy and Power Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Energy and Power Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7764-0530","authenticated-orcid":false,"given":"Meiyin","family":"Zhu","sequence":"additional","affiliation":[{"name":"Beihang Hangzhou Innovation Institute Yuhang, Hangzhou 310023, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shubo","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Energy and Power Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xitong","family":"Pei","sequence":"additional","affiliation":[{"name":"Research Institute of Aero-Engine, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhen","family":"Jiang","sequence":"additional","affiliation":[{"name":"School of Energy and Power Engineering, Beihang University, Beijing 100191, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1204","DOI":"10.1016\/j.cja.2019.01.017","article-title":"Two freedom linear parameter varying \u03bc synthesis control for flight environment testbed","volume":"32","author":"Zhu","year":"2019","journal-title":"Chin. J. Aeronaut."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"012144","DOI":"10.1088\/1742-6596\/1828\/1\/012144","article-title":"Two Degree-of-freedom \u03bc Synthesis Control for Turbofan Engine with Slow Actuator Dynamics and Uncertainties","volume":"1828","author":"Zhu","year":"2021","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Gu, N.N., Wang, X., and Lin, F.Q. (2019). Design of Disturbance Extended State Observer (D-ESO)-Based Constrained Full-State Model Predictive Controller for the Integrated Turbo-Shaft Engine\/Rotor System. Energies, 12.","DOI":"10.3390\/en12234496"},{"key":"ref_4","unstructured":"Dan, Z.H., Zhang, S., Bai, K.Q., Qian, Q.M., Pei, X.T., and Wang, X. (J. Propuls. Technol., 2020). Air Intake Environment Simulation of Altitude Test Facility Control Based on Extended State Observer, J. Propuls. Technol., in\u00a0press."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1016\/j.cja.2020.03.017","article-title":"Modified robust optimal adaptive control for flight environment simulation system with heat transfer uncertainty","volume":"34","author":"Zhu","year":"2021","journal-title":"Chin. J. Aeronaut."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Miao, K.Q., Wang, X., and Zhu, M.Y. (2020, January 21\u201325). Full Flight Envelope Transient Main Control Loop Design Based on LMI Optimization. Proceedings of the ASME Turbo Expo 2020, Virtual Online.","DOI":"10.1115\/GT2020-16048"},{"key":"ref_7","unstructured":"Gu, B.B. (2018). Robust Fuzzy Control for Aeroengines, Nanjing University of Aeronautics and Astronautics."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Amgad, M., Shakirah, M.T., Suliman, M.F., and Hitham, A. (2021). Deep-Learning Based Prognosis Approach for Remaining Useful Life Prediction of Turbofan Engine. Symmetry, 13.","DOI":"10.3390\/sym13101861"},{"key":"ref_9","unstructured":"Zhang, X.H., Liu, J.X., Li, M., Gen, J., and Song, Z.P. (J. Propuls. Technol., 2021). Fusion Control of Two Kinds of Control Schedules in Aeroengine Acceleration Process, J. Propuls. Technol., in\u00a0press."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yin, X., Shi, G., Peng, S., Zhang, Y., Zhang, B., and Su, W. (2022). Health State Prediction of Aero-Engine Gas Path System Considering Multiple Working Conditions Based on Time Domain Analysis and Belief Rule Base. Symmetry, 14.","DOI":"10.3390\/sym14010026"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MCS.2012.2214134","article-title":"Reinforcement learning and feedback control","volume":"32","author":"Frank","year":"2012","journal-title":"IEEE Control Syst. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_13","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_14","first-page":"1057","article-title":"Policy Gradient Methods for Reinforcement Learning with Function Approximation","volume":"12","author":"Richard","year":"2000","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_15","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 22\u201324). Deterministic Policy Gradient Algorithms. Proceedings of the International Conference on Machine Learning, Bejing, China."},{"key":"ref_16","unstructured":"Giulia, C., Shreyansh, D., and Roverto, C. (2021, January 15\u201317). Learning Transferable Policies for Autonomous Planetary Landing via Deep Reinforcement Learning. Proceedings of the ASCEND, Las Vegas, NV, USA."},{"key":"ref_17","unstructured":"Sun, D., Gao, D., Zheng, J.H., and Han, P. (J. Beijing Univ. Aeronaut. Astronaut., 2021). Reinforcement learning with demonstrations for UAV control, J. Beijing Univ. Aeronaut. Astronaut., in\u00a0press."},{"key":"ref_18","unstructured":"Kirk, H., and Steve, U. (2020, January 6\u201310). On Deep Reinforcement Learning for Spacecraft Guidance. Proceedings of the AIAA SciTech Forum, Orlando, FL, USA."},{"key":"ref_19","unstructured":"Hiroshi, K., Seiji, T., and Eiji, S. (2018, January 25\u201329). Feedback Control of Karman Vortex Shedding from a Cylinder using Deep Reinforcement Learning. Proceedings of the AIAA AVIATION Forum, Atlanta, GA, USA."},{"key":"ref_20","unstructured":"Hu, X. (2020). Design of Intelligent Controller for Variable Cycle Engine, Dalian University of Technology."},{"key":"ref_21","first-page":"1716","article-title":"Online Intelligent Optimization Algorithm for Adaptive Cycle Engine Performance","volume":"42","author":"Li","year":"2021","journal-title":"J. Propuls. Technol."},{"key":"ref_22","unstructured":"Wang, F. (2020). Research on Prediction of Civil Aero-Engine Gas Path Health State And Modeling Method of Spare Engine Allocation, Harbin Institute of Technology."},{"key":"ref_23","unstructured":"Li, Z. (2019). Research on Life-Cycle Maintenance Strategy Optimization of Civil Aeroengine Fleet, Harbin Institute of Technology."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Richter, H. (2013). Advanced Control of Turbofan Engines, National Defense Industry Press.","DOI":"10.1007\/978-1-4614-1171-0"},{"key":"ref_25","unstructured":"Miao, K.Q., Wang, X., and Zhu, M.Y. (J. Beijing Univ. Aeronaut. Astronaut., 2021). Dynamic Main Close-loop Control Optimal Design Based on LMI Method, J. Beijing Univ. Aeronaut. Astronaut., in\u00a0press."},{"key":"ref_26","unstructured":"Zeyan, P., Gang, L., Xingmin, G., and Yong, H. (2008). Principle of Aviation Gas Turbine, National Defense Industry Press."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/4\/684\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:43:34Z","timestamp":1760136214000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/4\/684"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,25]]},"references-count":26,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["sym14040684"],"URL":"https:\/\/doi.org\/10.3390\/sym14040684","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,25]]}}}