{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T23:48:59Z","timestamp":1771026539086,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2021,8,21]],"date-time":"2021-08-21T00:00:00Z","timestamp":1629504000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Guiding an aircraft to 4D waypoints at a certain heading is a multi-dimensional goal aircraft guidance problem. [d=Zu]In order to improve the performance and solve this problem, this paper proposes a multi-layer RL approach.To enhance the performance, in the present study, a multi-layer RL approach to solve the multi-dimensional goal aircraft guidance problem is proposed. The approach [d=Zu]enablesassists the autopilot in an ATC simulator to guide an aircraft to 4D waypoints at certain latitude, longitude, altitude, heading, and arrival time, respectively. To be specific, a multi-layer RL [d=Zu]approach is proposedmethod to simplify the neural network structure and reduce the state dimensions. A shaped reward function that involves the potential function and Dubins path method is applied. [d=Zu]Experimental and simulation results show that the proposed approachExperiments are conducted and the simulation results reveal that the proposed method can significantly improve the convergence efficiency and trajectory performance. [d=Zu]FurthermoreFurther, the results indicate possible application prospects in team aircraft guidance tasks, since the aircraft can directly approach a goal without waiting in a specific pattern, thereby overcoming the problem of current ATC simulators.<\/jats:p>","DOI":"10.3390\/s21165643","type":"journal-article","created":{"date-parts":[[2021,8,22]],"date-time":"2021-08-22T22:59:27Z","timestamp":1629673167000},"page":"5643","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Multi-Dimensional Goal Aircraft Guidance Approach Based on Reinforcement Learning with a Reward Shaping Algorithm"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2104-7233","authenticated-orcid":false,"given":"Wenqiang","family":"Zu","sequence":"first","affiliation":[{"name":"College of Computer Science, Sichuan University, Chengdu 610065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hongyu","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Computer Science, Sichuan University, Chengdu 610065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Renyu","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Computer Science, Sichuan University, Chengdu 610065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yulong","family":"Ji","sequence":"additional","affiliation":[{"name":"School of Aeronautics and Astronautics, Sichuan University, Chengdu 610065, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,21]]},"reference":[{"key":"ref_1","unstructured":"Dunn, C., Valasek, J., and Kirkpatrick, K.C. (2012). Unmanned Air System Search and Localization Guidance Using Reinforcement Learning, Infotech@ Aerospace."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"768","DOI":"10.1134\/S1064230720050123","article-title":"Linear-cubic locally optimal control of linear systems and its application for aircraft guidance","volume":"59","author":"Verba","year":"2020","journal-title":"J. Comput. Syst. Sci. Int."},{"key":"ref_3","first-page":"1","article-title":"System Identification Guidance for Multirotor Aircraft: Dynamic Scaling and Test Techniques","volume":"66","author":"Ivler","year":"2021","journal-title":"J. Am. Helicopter Soc."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1109\/TAES.2020.3046328","article-title":"Cooperative active aircraft protection guidance using line-of-sight approach","volume":"57","author":"Kumar","year":"2020","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Morani, G., Di Vito, V., Corraro, F., Grevtsov, N., and Dymchenko, A. (2013, January 19\u201322). Automatic Guidance through 4D Waypoints with time and spatial margins. Proceedings of the AIAA Guidance, Navigation, and Control (GNC) Conference, Boston, MA, USA.","DOI":"10.2514\/6.2013-4892"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1134\/S1064230721010123","article-title":"Optimization of automatic support systems of air objects based on local quadratic-biquadratic functionals. I. Synthesis of optimum control","volume":"60","author":"Verba","year":"2021","journal-title":"J. Comput. Syst. Sci. Int."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1116","DOI":"10.2514\/1.G003791","article-title":"Stability analysis for incremental nonlinear dynamic inversion control","volume":"42","author":"Wang","year":"2019","journal-title":"J. Guid. Control. Dyn."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"474","DOI":"10.1016\/j.ast.2018.12.030","article-title":"A visual\/inertial integrated landing guidance method for UAV landing on the ship","volume":"85","author":"Meng","year":"2019","journal-title":"Aerosp. Sci. Technol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"134668","DOI":"10.1109\/ACCESS.2020.3010963","article-title":"A hybrid CNN-LSTM model for aircraft 4D trajectory prediction","volume":"8","author":"Ma","year":"2020","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Juntama, P., Chaimatanan, S., Alam, S., and Delahaye, D. (2020, January 3\u20134). A Distributed Metaheuristic Approach for Complexity Reduction in Air Traffic for Strategic 4D Trajectory Optimization. Proceedings of the 2020 International Conference on Artificial Intelligence and Data Analytics for Air Transportation (AIDA-AT), Singapore.","DOI":"10.1109\/AIDA-AT48540.2020.9049200"},{"key":"ref_11","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_12","unstructured":"Vonk, B. (2021, August 06). Exploring Reinforcement Learning Methods for Autonomous Sequencing and Spacing of Aircraft. Available online: https:\/\/repository.tudelft.nl\/islandora\/object\/uuid:2e776b60-cd4e-4268-93e3-3fcc81cd794f."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wang, Z., Li, H., Wu, H., Shen, F., and Lu, R. (2018, January 8\u20139). Design of agent training environment for aircraft landing guidance based on deep reinforcement learning. Proceedings of the 2018 11th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China.","DOI":"10.1109\/ISCID.2018.10118"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1007\/s10846-017-0696-1","article-title":"Learning to perform a perched landing on the ground using deep reinforcement learning","volume":"92","author":"Waldock","year":"2018","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.neucom.2020.02.008","article-title":"Principled reward shaping for reinforcement learning via lyapunov stability theory","volume":"393","author":"Dong","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zou, H., Ren, T., Yan, D., Su, H., and Zhu, J. (2021, January 2\u20139). Learning Task-Distribution Reward Shaping with Meta-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v35i12.17337"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"du Preez-Wilkinson, N., and Gallagher, M. (2020). Fitness Landscape Features and Reward Shaping in Reinforcement Learning Policy Spaces. International Conference on Parallel Problem Solving from Nature, Springer.","DOI":"10.1007\/978-3-030-58115-2_35"},{"key":"ref_18","unstructured":"Levy, A., Konidaris, G., Platt, R., and Saenko, K. (2017). Learning multi-level hierarchies with hindsight. arXiv, Available online: https:\/\/arxiv.org\/abs\/1712.00948."},{"key":"ref_19","unstructured":"Brittain, M., and Wei, P. (2021, August 06). Autonomous Aircraft Sequencing and Separation with Hierarchical Deep Reinforcement Learning. Available online: https:\/\/lib.dr.iastate.edu\/cgi\/viewcontent.cgi?article=9470&context=etd#page=91."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.trc.2013.06.010","article-title":"Reward functions for learning to control in air traffic flow management","volume":"35","author":"Cruciol","year":"2013","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1729881421989546","DOI":"10.1177\/1729881421989546","article-title":"A pretrained proximal policy optimization algorithm with reward shaping for aircraft guidance to a moving destination in three-dimensional continuous space","volume":"18","author":"Wang","year":"2021","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Radac, M.B., and Borlea, A.I. (2021). Virtual State Feedback Reference Tuning and Value Iteration Reinforcement Learning for Unknown Observable Systems Control. Energies, 14.","DOI":"10.3390\/en14041006"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"42568","DOI":"10.1109\/ACCESS.2021.3062457","article-title":"A novel hierarchical soft actor\u2013critic algorithm for multi-logistics robots task allocation","volume":"9","author":"Tang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, T., Yang, D., Xie, X., and Zhang, H. (2021). Event-triggered control of nonlinear discrete-time system with unknown dynamics based on HDP (\u03bb). IEEE Trans. Cybern.","DOI":"10.1109\/TCYB.2020.3044595"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Manyam, S.G., Casbeer, D., Von Moll, A.L., and Fuchs, Z. (2019, January 7\u201311). Shortest Dubins path to a circle. Proceedings of the AIAA Scitech 2019 Forum, San Diego, CA, USA.","DOI":"10.2514\/6.2019-0919"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhou, Y., Zhou, W., Fei, M., and Wang, S. (2020). 3D Curve Planning Algorithm of Aircraft Under Multiple Constraints. Recent Featured Applications of Artificial Intelligence Methods, Springer. LSMS 2020 and ICSEE 2020 Workshops.","DOI":"10.1007\/978-981-33-6378-6_18"},{"key":"ref_27","unstructured":"Ku\u010derov\u00e1, K., V\u00e1\u0148, P., and Faigl, J. (April, January 30). On finding time-efficient trajectories for fixed-wing aircraft using dubins paths with multiple radii. Proceedings of the 35th Annual ACM Symposium on Applied Computing, Brno, Czech Republic."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Van Otterlo, M., and Wiering, M. (2012). Reinforcement learning and markov decision processes. Reinforcement Learning, Springer.","DOI":"10.1007\/978-3-642-27645-3_1"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2017","DOI":"10.1162\/089976699300016070","article-title":"A unified analysis of value-function-based reinforcement-learning algorithms","volume":"11","author":"Littman","year":"1999","journal-title":"Neural Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"103919","DOI":"10.1016\/j.engappai.2020.103919","article-title":"Policy-based reinforcement learning for time series anomaly detection","volume":"95","author":"Yu","year":"2020","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Brittain, M., and Wei, P. (2019, January 27\u201330). Autonomous separation assurance in an high-density en route sector: A deep multi-agent reinforcement learning approach. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917217"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/0893-6080(91)90005-P","article-title":"A Gaussian potential function network with hierarchically self-organizing learning","volume":"4","author":"Lee","year":"1991","journal-title":"Neural Netw."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1016\/j.robot.2005.11.004","article-title":"Visual navigation and obstacle avoidance using a steering potential function","volume":"54","author":"Huang","year":"2006","journal-title":"Robot. Auton. Syst."},{"key":"ref_34","unstructured":"Hoekstra, J.M., and Ellerbroek, J. (2016, January 20\u201324). Bluesky ATC simulator project: An open data and open source approach. Proceedings of the 7th International Conference on Research in Air Transportation, Philadelphia, PA, USA."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Sun, J., Hoekstra, J.M., and Ellerbroek, J. (2020). OpenAP: An open-source aircraft performance model for air transportation studies and simulations. Aerospace, 7.","DOI":"10.3390\/aerospace7080104"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Hara, K., Saito, D., and Shouno, H. (2015, January 12\u201317). Analysis of function of rectified linear unit used in deep learning. Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland.","DOI":"10.1109\/IJCNN.2015.7280578"},{"key":"ref_37","unstructured":"Agostinelli, F., Hoffman, M., Sadowski, P., and Baldi, P. (2014). Learning activation functions to improve deep neural networks. arXiv, Available online: https:\/\/arxiv.org\/abs\/1412.6830."},{"key":"ref_38","unstructured":"Gao, B., and Pavel, L. (2017). On the properties of the softmax function with application in game theory and reinforcement learning. arXiv, Available online: https:\/\/arxiv.org\/abs\/1704.00805."},{"key":"ref_39","unstructured":"Kakade, S.M. (2003). On the Sample Complexity of Reinforcement Learning, University of London, University College London."},{"key":"ref_40","unstructured":"Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2017, January 4\u20139). Hindsight experience replay. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/16\/5643\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:48:50Z","timestamp":1760165330000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/16\/5643"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,21]]},"references-count":40,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["s21165643"],"URL":"https:\/\/doi.org\/10.3390\/s21165643","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,21]]}}}