{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,19]],"date-time":"2026-07-19T09:46:23Z","timestamp":1784454383574,"version":"3.55.0"},"reference-count":45,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2023,5,28]],"date-time":"2023-05-28T00:00:00Z","timestamp":1685232000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2020YFB1807500"],"award-info":[{"award-number":["2020YFB1807500"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["SKX212010039"],"award-info":[{"award-number":["SKX212010039"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"name":"54th Research Institute of China Electronics Technology Group Corporation Cooperative Project","award":["2020YFB1807500"],"award-info":[{"award-number":["2020YFB1807500"]}]},{"name":"54th Research Institute of China Electronics Technology Group Corporation Cooperative Project","award":["SKX212010039"],"award-info":[{"award-number":["SKX212010039"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Due to the rapid development of the space\u2013air\u2013ground integrated network (SAGIN), a satellite communication system has the advantages of wide coverage and low requirements for a geographical environment and is gradually becoming the main competitive technology for 6G. The low-earth-orbit (LEO) satellite network has the characteristics of low transmission delay, small propagation loss, and global coverage, and its exploration has become the main research object of contemporary satellite communications. However, traditional routing algorithms cannot adapt to the characteristics of the high dynamics and load-balancing requirements of LEO satellite networks. In this paper, a load-balancing routing algorithm for LEO satellites based on Deep Q-Network (DQN-LLRA) is proposed by using deep reinforcement learning. Making use of the model obtained by the DQN training, satellite nodes can select the best routing results according to the delay, bandwidth, and queue utilization of the surrounding satellite nodes. The simulation and analysis show that the path load obtained by the proposed algorithm is low. Compared with the Q-learning-based algorithm, this algorithm reduces the maximum queue utilization rate of the routing path by 5%, reduces the average queue utilization rate of the routing path by 13%, and effectively balances the load in the network.<\/jats:p>","DOI":"10.3390\/rs15112801","type":"journal-article","created":{"date-parts":[[2023,5,28]],"date-time":"2023-05-28T15:29:52Z","timestamp":1685287792000},"page":"2801","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["DRL-Based Load-Balancing Routing Scheme for 6G Space\u2013Air\u2013Ground Integrated Networks"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9668-1255","authenticated-orcid":false,"given":"Feihu","family":"Dong","sequence":"first","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"},{"name":"The 54th Research Institute of China Electronics Technology Group Corporation, Shijiazhuang 050051, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2325-3831","authenticated-orcid":false,"given":"Jiaxin","family":"Song","sequence":"additional","affiliation":[{"name":"School of Telecommunications Engineering, Xidian University, Xi\u2019an 710071, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasheng","family":"Zhang","sequence":"additional","affiliation":[{"name":"The 54th Research Institute of China Electronics Technology Group Corporation, Shijiazhuang 050051, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuqi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Telecommunications Engineering, Xidian University, Xi\u2019an 710071, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3545-1122","authenticated-orcid":false,"given":"Tao","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1002\/sat.714","article-title":"Performance evaluation of adaptive routing algorithms in packet-switched intersatellite link networks","volume":"20","author":"Mohori","year":"2002","journal-title":"Int. J. Satell. Commun. Netw."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2342","DOI":"10.1109\/JIOT.2020.3048345","article-title":"Blockchain-Enabled Secure Data Sharing Scheme in Mobile-Edge Computing: An Asynchronous Advantage Actor\u2013Critic Learning Approach","volume":"8","author":"Liu","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1109\/TNSE.2022.3207214","article-title":"Aerial Edge Computing on Orbit: A Task Offloading and Allocation Scheme","volume":"10","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chen, C., Wang, C., Liu, B., He, C., Cong, L., and Wan, S. (2023). Edge Intelligence Empowered Vehicle Detection and Image Segmentation for Autonomous Vehicles. IEEE Trans. Intell. Transp. Syst., 1\u201312.","DOI":"10.1109\/TITS.2022.3232153"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"5186","DOI":"10.1109\/TITS.2023.3241251","article-title":"A Cooperative Vehicle-Infrastructure System for Road Hazards Detection With Edge Intelligence","volume":"24","author":"Chen","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"474","DOI":"10.1177\/0272989X09353194","article-title":"Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty","volume":"30","author":"Alagoz","year":"2010","journal-title":"Med. Decis. Mak."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_8","first-page":"553","article-title":"Modelling and Performance Analysis of 2.5 Gbps Inter-satellite Optical Wireless Communication (IsOWC) System in LEO Constellation","volume":"13","author":"Kavas","year":"2018","journal-title":"J. Commun."},{"key":"ref_9","unstructured":"van der Ha, J.C. Launching and Building the IRIDIUM\u00ae Constellation. Proceedings of the Mission Design & Implementation of Satellite Constellations."},{"key":"ref_10","unstructured":"Henderson, T., and Katz, R. (December, January 27). On distributed, geographic-based packet routing for LEO satellite networks. Proceedings of the Globecom\u201900-IEEE. Global Telecommunications Conference. Conference Record (Cat. No.00CH37137), San Francisco, CA, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1109\/JSAC.2003.819974","article-title":"Routing in ISL networks considering empirical IP traffic","volume":"22","author":"Svigelj","year":"2004","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_12","first-page":"2169","article-title":"Mobility-Aware Multi-Hop Task Offloading for Autonomous Driving in Vehicular Edge Computing and Networks","volume":"24","author":"Liu","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_13","unstructured":"Gounder, V., Prakash, R., and Abu-Amara, H. (1999, January 12\u201313). Routing in LEO-based satellite networks. Proceedings of the 1999 IEEE Emerging Technologies Symposium. Wireless Communications and Systems (IEEE Cat. No.99EX297), Richardson, TX, USA."},{"key":"ref_14","unstructured":"Long, H., Shen, Y., Guo, M., and Tang, F. (2013, January 25\u201328). LABERIO: Dynamic load-balanced Routing in OpenFlow-enabled Networks. Proceedings of the 2013 IEEE 27th International Conference on Advanced Information Networking and Applications (AINA), Barcelona, Spain."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"902","DOI":"10.1109\/TAES.2002.1039407","article-title":"Routing in networks of intersatellite links","volume":"38","author":"Franck","year":"2002","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1109\/TWC.2002.804176","article-title":"Adaptive routing for packet-oriented intersatellite link networks: Performance in various traffic scenarios","volume":"1","author":"Mohorcic","year":"2002","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1007\/s11276-006-0733-7","article-title":"Minimum flow maximum residual routing in LEO satellite networks using routing set","volume":"14","author":"Kucukates","year":"2008","journal-title":"Wirel. Netw."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Cigliano, A., and Zampognaro, F. (2020, January 25\u201327). A Machine Learning approach for routing in satellite Mega-Constellations. Proceedings of the 2020 International Symposium on Advanced Electrical and Communication Technologies (ISAECT), Virtual.","DOI":"10.1109\/ISAECT50560.2020.9523672"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wang, H., Ran, Y., Zhao, L., Wang, J., Luo, J., and Zhang, T. (2021, January 25\u201327). GRouting: Dynamic Routing for LEO Satellite Networks with Graph-based Deep Reinforcement Learning. Proceedings of the 2021 4th International Conference on Hot Information-Centric Networking (HotICN), Nanjing, China.","DOI":"10.1109\/HotICN53262.2021.9680855"},{"key":"ref_20","unstructured":"Kucukates, R., and Ersoy, C. (July, January 30). High performance routing in a LEO satellite network. Proceedings of the Eighth IEEE Symposium on Computers and Communications, Kemer-Antalya, Turkey."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"112044","DOI":"10.1109\/ACCESS.2019.2934932","article-title":"Load-Balancing Routing Algorithm Based on Segment Routing for Traffic Return in LEO Satellite Networks","volume":"7","author":"Liu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ju, Y., Zou, G., Bai, H., Liu, L., Pei, Q., Wu, C., and Otaibi, S.A. (2023). Random Beam Switching: A Physical Layer Key Generation Approach to Safeguard mmWave Electronic Devices. IEEE Trans. Consum. Electron., 1.","DOI":"10.1109\/TCE.2023.3273125"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Taleb, T., Mashimo, D., Jamalipour, A., Hashimoto, K., Nemoto, Y., and Kato, N. (December, January 27). SAT04-3: ELB: An Explicit Load Balancing Routing Protocol for Multi-Hop NGEO Satellite Constellations. Proceedings of the IEEE Globecom 2006, San Francisco, CA, USA.","DOI":"10.1109\/GLOCOM.2006.523"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1109\/TNET.2008.918084","article-title":"Explicit Load Balancing Technique for NGEO Satellite IP Networks With On-Board Processing Capabilities","volume":"17","author":"Taleb","year":"2009","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3380","DOI":"10.1109\/TWC.2014.041014.130040","article-title":"TLR: A Traffic-Light-Based Intelligent Routing Strategy for NGEO Satellite IP Networks","volume":"13","author":"Song","year":"2014","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"155136","DOI":"10.1109\/ACCESS.2020.3017615","article-title":"A Load Balancing Routing Strategy for LEO Satellite Network","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"107166","DOI":"10.1016\/j.comnet.2020.107166","article-title":"An optimal delay routing algorithm considering delay variation in the LEO satellite communication network","volume":"173","author":"Geng","year":"2020","journal-title":"Comput. Netw."},{"key":"ref_28","unstructured":"Wei, S., Cheng, H., Liu, M., and Ren, M. (2017, January 10\u201311). Optimal Strategy Routing in LEO Satellite Network Based on Cooperative Game Theory. Proceedings of the Space Information Networks: Second International Conference, SINC 2017, Yinchuan, China."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1007\/s11276-016-1326-8","article-title":"A QoS routing strategy using fuzzy logic for NGEO satellite IP networks","volume":"24","author":"Jiang","year":"2018","journal-title":"Wirel. Netw."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Pan, T., Huang, T., Li, X., Chen, Y., Xue, W., and Liu, Y. (2019, January 20\u201324). OPSPF: Orbit Prediction Shortest Path First Routing for Resilient LEO Satellite Networks. Proceedings of the ICC 2019\u20132019 IEEE International Conference on Communications (ICC), Shanghai, China.","DOI":"10.1109\/ICC.2019.8761611"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hao, L., Ren, P., and Du, Q. (2020, January 21\u201323). Satellite QoS Routing Algorithm Based on Energy Aware and Load Balancing. Proceedings of the 2020 International Conference on Wireless Communications and Signal Processing (WCSP), Nanjing, China.","DOI":"10.1109\/WCSP49889.2020.9299827"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zuo, P., Wang, C., Wei, Z., Li, Z., Zhao, H., and Jiang, H. (2022, January 19\u201322). Deep Reinforcement Learning Based Load Balancing Routing for LEO Satellite Network. Proceedings of the 2022 IEEE 95th Vehicular Technology Conference: (VTC2022-Spring), Helsinki, Finland.","DOI":"10.1109\/VTC2022-Spring54318.2022.9860582"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Xu, Q., Zhang, Y., Wu, K., Wang, J., and Lu, K. (2019, January 4\u20137). Evaluating and Boosting Reinforcement Learning for Intra-Domain Routing. Proceedings of the 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems (MASS), Monterey, CA, USA.","DOI":"10.1109\/MASS.2019.00039"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5555","DOI":"10.1109\/TITS.2023.3242997","article-title":"Joint Secure Offloading and Resource Allocation for Vehicular Edge Computing Network: A Multi-Agent Deep Reinforcement Learning Approach","volume":"24","author":"Ju","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"66384","DOI":"10.1109\/ACCESS.2019.2917564","article-title":"Q-Learning Based Content Placement Method for Dynamic Cloud Content Delivery Networks","volume":"7","author":"Liu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"9668","DOI":"10.1109\/JIOT.2019.2930459","article-title":"A Q-Learning Based Framework for Congested Link Identification","volume":"6","author":"Pan","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1016\/j.comnet.2019.06.006","article-title":"A Q-learning algorithm for task scheduling based on improved SVM in wireless sensor networks","volume":"161","author":"Wei","year":"2019","journal-title":"Comput. Netw."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1109\/JIOT.2019.2945640","article-title":"Deep Reinforcement Learning for Cooperative Content Caching in Vehicular Edge Computing and Networks","volume":"7","author":"Qiao","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Tu, Z., Zhou, H., Li, K., Li, G., and Shen, Q. (2019, January 9\u201313). A Routing Optimization Method for Software-Defined SGIN Based on Deep Reinforcement Learning. Proceedings of the 2019 IEEE Globecom Workshops (GC Wkshps), Big Island, HI, USA.","DOI":"10.1109\/GCWkshps45667.2019.9024680"},{"key":"ref_40","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N.M.O., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Ju, Y., Yang, M., Chakraborty, C., Liu, L., Pei, Q., Xiao, M., and Yu, K. (2023). Reliability-Security Tradeoff Analysis in mmWave Ad Hoc Based CPS. ACM Trans. Sens. Netw.","DOI":"10.1145\/3582556"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1997","DOI":"10.1109\/TCOMM.2023.3240754","article-title":"Deep Reinforcement Learning Based Joint Beam Allocation and Relay Selection in mmWave Vehicular Networks","volume":"71","author":"Ju","year":"2023","journal-title":"IEEE Trans. Commun."},{"key":"ref_43","unstructured":"Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017, January 4\u20139). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. Proceedings of the 31st International Conference on Neural Information Processing Systems NIPS\u201917, Long Beach, CA, USA."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Qin, Z., Yao, H., and Mai, T. (2020, January 15\u201319). Traffic Optimization in Satellites Communications: A Multi-agent Reinforcement Learning Approach. Proceedings of the 2020 International Wireless Communications and Mobile Computing (IWCMC), Limassol, Cyprus.","DOI":"10.1109\/IWCMC48107.2020.9148523"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"3759631","DOI":"10.1155\/2021\/3759631","article-title":"Reinforcement Learning-Based Routing Algorithm in Satellite-Terrestrial Integrated Networks","volume":"2021","author":"Yin","year":"2021","journal-title":"Wirel. Commun. Mob. Comput."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/11\/2801\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:43:44Z","timestamp":1760125424000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/11\/2801"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,28]]},"references-count":45,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["rs15112801"],"URL":"https:\/\/doi.org\/10.3390\/rs15112801","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,28]]}}}