{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T16:59:57Z","timestamp":1791478797787,"version":"4.3.4"},"reference-count":28,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2023,5,29]],"date-time":"2023-05-29T00:00:00Z","timestamp":1685318400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Fast convergence routing is a critical issue for Low Earth Orbit (LEO) constellation networks because these networks have dynamic topology changes, and transmission requirements can vary over time. However, most of the previous research has focused on the Open Shortest Path First (OSPF) routing algorithm, which is not well-suited to handle the frequent changes in the link state of the LEO satellite network. In this regard, we propose a Fast-Convergence Reinforcement Learning Satellite Routing Algorithm (FRL\u2013SR) for LEO satellite networks, where the satellite can quickly obtain the network link status and adjust its routing strategy accordingly. In FRL\u2013SR, each satellite node is considered an agent, and the agent selects the appropriate port for packet forwarding based on its routing policy. Whenever the satellite network state changes, the agent sends \u201chello\u201d packets to the neighboring nodes to update their routing policy. Compared to traditional reinforcement learning algorithms, FRL\u2013SR can perceive network information faster and converge faster. Additionally, FRL\u2013SR can mask the dynamics of the satellite network topology and adaptively adjust the forwarding strategy based on the link state. The experimental results demonstrate that the proposed FRL\u2013SR algorithm outperforms the Dijkstra algorithm in the performance of average delay, packet arriving ratio, and network load balance.<\/jats:p>","DOI":"10.3390\/s23115180","type":"journal-article","created":{"date-parts":[[2023,5,30]],"date-time":"2023-05-30T02:33:27Z","timestamp":1685414007000},"page":"5180","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Fast-Convergence Reinforcement Learning for Routing in LEO Satellite Networks"],"prefix":"10.3390","volume":"23","author":[{"given":"Zhaolong","family":"Ding","sequence":"first","affiliation":[{"name":"School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China"},{"name":"Innovation Academy for Microsatellites of CAS, Shanghai 201304, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huijie","family":"Liu","sequence":"additional","affiliation":[{"name":"Innovation Academy for Microsatellites of CAS, Shanghai 201304, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Feng","family":"Tian","sequence":"additional","affiliation":[{"name":"Innovation Academy for Microsatellites of CAS, Shanghai 201304, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0351-853X","authenticated-orcid":false,"given":"Zijian","family":"Yang","sequence":"additional","affiliation":[{"name":"Innovation Academy for Microsatellites of CAS, Shanghai 201304, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nan","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Tsai, K.-C., Fan, L., Wang, L.-C., Lent, R., and Han, Z. (2022, January 10\u201313). Multi-Commodity Flow Routing for Large-Scale LEO Satellite Networks Using Deep Reinforcement Learning. Proceedings of the 2022 IEEE Wireless Communications and Networking Conference (WCNC), Austin, TX, USA.","DOI":"10.1109\/WCNC51071.2022.9771734"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/MWC.2019.1800299","article-title":"Broadband LEO Satellite Communications: Architectures and Key Technologies","volume":"26","author":"Su","year":"2019","journal-title":"IEEE Wirel. Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2872","DOI":"10.1109\/TNSE.2020.3039499","article-title":"DRL-ER: An Intelligent Energy-Aware Routing Protocol With Guaranteed Delay Bounds in Satellite Mega-Constellations","volume":"8","author":"Liu","year":"2021","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1109\/35.601743","article-title":"QoS guarantees for multimedia services on a TDMA-based satellite network","volume":"35","author":"Mauger","year":"1997","journal-title":"IEEE Commun. Mag."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1636","DOI":"10.1109\/49.634801","article-title":"A dynamic routing concept for ATM-based satellite personal communication networks","volume":"15","author":"Werner","year":"1997","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Cao, X., Li, Y., Xiong, X., and Wang, J. (2022). Dynamic Routings in Satellite Networks: An Overview. Sensors, 22.","DOI":"10.3390\/s22124552"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6009","DOI":"10.1109\/TVT.2019.2910570","article-title":"Temporal Netgrid Model-Based Dynamic Routing in Large-Scale Small Satellite Networks","volume":"68","author":"Li","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1109\/90.917071","article-title":"A distributed routing algorithm for datagram traffic in LEO satellite networks","volume":"9","author":"Ekici","year":"2001","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Liu, X., Yan, X., Jiang, Z., Li, C., and Yang, Y. (2015, January 6\u20139). A low-complexity routing algorithm based on load balancing for LEO satellite networks. Proceedings of the 2015 IEEE 82nd Vehicular Technology Conference (VTC2015-Fall), Boston, MA, USA.","DOI":"10.1109\/VTCFall.2015.7390795"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Guo, A., Zhao, C., Xu, F., and Qiu, C. (2017, January 25\u201327). LEO satellite routing algorithm in software defined space terrestrial integrated network. Proceedings of the 2017 17th International Symposium on Communications and Information Technologies (ISCIT), Cairns, Australia.","DOI":"10.1109\/ISCIT.2017.8261222"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"19823","DOI":"10.1109\/ACCESS.2018.2820719","article-title":"scMPTCP: SDN Cooperated Multipath Transfer for Satellite Network With Load Awareness","volume":"6","author":"Jiang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1109\/CC.2018.8438270","article-title":"The impact of delay in software-defined integrated terrestrial-satellite networks","volume":"15","author":"Boero","year":"2018","journal-title":"China Commun."},{"key":"ref_13","unstructured":"Jay, N., Rotman, N., Godfrey, B., Schapira, M., and Tamar, A. (2019, January 9\u201315). A deep reinforcement learning perspective on internet congestion control. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mao, H., Netravali, R., and Alizadeh, M. (2017, January 21\u201325). Neural adaptive video streaming with pensieve. Proceedings of the Conference of the ACM Special Interest Group on Data Communication, Los Angeles, CA, USA.","DOI":"10.1145\/3098822.3098843"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1109\/TWC.2017.2769644","article-title":"User scheduling and resource allocation in hetnets with hybrid energy supply: An actor-critic reinforcement learning approach","volume":"17","author":"Wei","year":"2018","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, X., Dai, Z., and Xu, Z. (2021, January 7\u201310). LEO Satellite Network Routing Algorithm Based on Reinforcement Learning. Proceedings of the 2021 IEEE 4th International Conference on Electronics Technology (ICET), Chengdu, China.","DOI":"10.1109\/ICET51757.2021.9451072"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4945","DOI":"10.1109\/JLT.2022.3175865","article-title":"Deep Reinforcement Learning-Based Routing and Spectrum Assignment of EONs by Exploiting GCN and RNN for Feature Extraction","volume":"40","author":"Xu","year":"2022","journal-title":"J. Light. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1016\/j.cja.2022.06.021","article-title":"Reinforcement learning based dynamic distributed routing scheme for mega LEO satellite networks","volume":"36","author":"Huang","year":"2023","journal-title":"Chin. J. Aeronaut."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2432","DOI":"10.1109\/COMST.2017.2707140","article-title":"State-of-the-art deep learning: Evolving machine intelligence toward tomorrow\u2019s intelligent network traffic control systems","volume":"19","author":"Fadlullah","year":"2017","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_20","unstructured":"Chang, H.S., Kim, B.W., Lee, C.G., Choi, Y., Min, S.L., Yang, H.S., and Kim, C.S. (1995, January 14\u201316). Topological design and routing for low-Earth orbit satellite networks. Proceedings of the GLOBECOM\u201995, Singapore."},{"key":"ref_21","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1109\/JSTSP.2021.3055957","article-title":"Tensor-based reinforcement learning for network routing","volume":"15","author":"Zhuang","year":"2021","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Li, X., Hu, X., Li, W., and Hu, H. (2019, January 20\u201324). A Multi-Agent Reinforcement Learning Routing Protocol for Underwater Optical Sensor Networks. Proceedings of the ICC 2019\u20142019 IEEE International Conference on Communications (ICC), Shanghai, China.","DOI":"10.1109\/ICC.2019.8761441"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Liu, Y., Jiang, Z., Zhang, S., and Xu, S. (2020, January 7\u201311). Deep Reinforcement Learning-Based Beam Tracking for Low-Latency Services in Vehicular Networks. Proceedings of the ICC 2020\u20142020 IEEE International Conference on Communications (ICC), Dublin, Ireland.","DOI":"10.1109\/ICC40277.2020.9148759"},{"key":"ref_25","unstructured":"Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017, January 4\u20139). Multi-agent actor-critic for mixed cooperative-competitive environments. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2258","DOI":"10.1109\/LCOMM.2017.2656879","article-title":"QGeo: Q-learning-based geographic ad hoc routing protocol for unmanned robotic networks","volume":"21","author":"Jung","year":"2017","journal-title":"IEEE Commun. Lett."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2022\/8929437","article-title":"LEO laser microwave hybrid inter-satellite routing strategy based on modified Q-routing algorithm","volume":"2022","author":"Zheng","year":"2022","journal-title":"EURASIP J. Wirel. Commun. Netw."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/TMC.2018.2831679","article-title":"Multipath Cooperative Routing with Efficient Acknowledgement for LEO Satellite Networks","volume":"18","author":"Tang","year":"2019","journal-title":"IEEE Trans. Mob. Comput."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/11\/5180\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:44:41Z","timestamp":1760125481000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/11\/5180"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,29]]},"references-count":28,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23115180"],"URL":"https:\/\/doi.org\/10.3390\/s23115180","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,29]]}}}