{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T15:29:01Z","timestamp":1771514941812,"version":"3.50.1"},"reference-count":29,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,3,30]],"date-time":"2023-03-30T00:00:00Z","timestamp":1680134400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The HORIZON-CL4-2021-SPACE-01 project","award":["101081983"],"award-info":[{"award-number":["101081983"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>In non-terrestrial networks, where low Earth orbit satellites and user equipment move relative to each other, line-of-sight tracking and adapting to channel state variations due to endpoint movements are a major challenge. Therefore, continuous line-of-sight estimation and channel impairment compensation are crucial for user equipment to access a satellite and maintain connectivity. In this paper, we propose a framework based on actor-critic reinforcement learning for traffic scheduling in non-terrestrial networks scenario where the channel state is non-stationary due to the variability of the line of sight, which depends on the current satellite elevation. We deploy the framework as an agent in a multipath routing scheme where the user equipment can access more than one satellite simultaneously to improve link reliability and throughput. We investigate how the agent schedules traffic in multiple satellite links by adopting policies that are evaluated by an actor-critic reinforcement learning approach. The agent continuously trains its model based on variations in satellite elevation angles, handovers, and relative line-of-sight probabilities. We compare the agent\u2019s retraining time with the satellite visibility intervals to investigate the effectiveness of the agent\u2019s learning rate. We carry out performance analysis while considering the dense urban area of Paris, where high-rise buildings significantly affect the line of sight. The simulation results show how the learning agent selects the scheduling policy when it is connected to a pair of satellites. The results also show that the retraining time of the learning agent is up to 0.1times the satellite visibility time at given elevations, which guarantees efficient use of satellite visibility.<\/jats:p>","DOI":"10.3390\/rs15071842","type":"journal-article","created":{"date-parts":[[2023,3,30]],"date-time":"2023-03-30T04:45:35Z","timestamp":1680151535000},"page":"1842","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Learning-Based Traffic Scheduling in Non-Stationary Multipath 5G Non-Terrestrial Networks"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0260-2465","authenticated-orcid":false,"given":"Achilles","family":"Machumilane","sequence":"first","affiliation":[{"name":"Institute of Information Science and Technologies (ISTI), CNR, 56124 Pisa, Italy"},{"name":"Department of Information Engineering, University of Pisa, 56126 Pisa, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8134-7844","authenticated-orcid":false,"given":"Alberto","family":"Gotta","sequence":"additional","affiliation":[{"name":"Institute of Information Science and Technologies (ISTI), CNR, 56124 Pisa, Italy"},{"name":"CNIT\u2014National Inter-University Consortium for Telecommunications, 43124 Parma, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3704-4133","authenticated-orcid":false,"given":"Pietro","family":"Cassar\u00e1","sequence":"additional","affiliation":[{"name":"Institute of Information Science and Technologies (ISTI), CNR, 56124 Pisa, Italy"},{"name":"CNIT\u2014National Inter-University Consortium for Telecommunications, 43124 Parma, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0171-4315","authenticated-orcid":false,"given":"Giuseppe","family":"Amato","sequence":"additional","affiliation":[{"name":"Institute of Information Science and Technologies (ISTI), CNR, 56124 Pisa, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3715-149X","authenticated-orcid":false,"given":"Claudio","family":"Gennaro","sequence":"additional","affiliation":[{"name":"Institute of Information Science and Technologies (ISTI), CNR, 56124 Pisa, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,30]]},"reference":[{"key":"ref_1","unstructured":"Bacco, M., Davoli, F., Giambene, G., Gotta, A., Luglio, M., Marchese, M., Patrone, F., and Roseti, C. (October, January 30). Networking Challenges for Non-Terrestrial Networks Exploitation in 5G. Proceedings of the IEEE 2nd 5G World Forum (5GWF), Dresden, Germany."},{"key":"ref_2","unstructured":"3GPP (2023, January 05). Technical Specification Group Radio Access Network; Solutions for NR to Support Non-Terrestrial Networks (NTN): TR 38.821 V16.1.0 (2021-05), (Release 16). Available online: https:\/\/portal.3gpp.org\/desktopmodules\/Specifications\/SpecificationDetails.aspx?specificationId=3525."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/MCOM.001.2100904","article-title":"A Path-Aware Scheduler for Air-to-Ground Multipath Multimedia Delivery in Real Time","volume":"60","author":"Machumilane","year":"2022","journal-title":"IEEE Commun. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bacco, M., Cassar\u00e1, P., Gotta, A., and Pellegrini, V. (2019, January 22\u201325). Real-Time Multipath Multimedia Traffic in Cellular Networks for Command and Control Applications. Proceedings of the 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.","DOI":"10.1109\/VTCFall.2019.8891090"},{"key":"ref_5","unstructured":"Recommendation, I. (2017). Propagation Data Required for The design of Earth-Space Land Mobile Telecommunication Systems, International Telecommunication Union."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Paasch, C., Ferlin, S., Alay, O., and Bonaventure, O. (2014, January 18). Experimental Evaluation of Multipath TCP Schedulers. Proceedings of the ACM SIGCOMM Workshop on Capacity Sharing Workshop, Chicago, IL, USA.","DOI":"10.1145\/2630088.2631977"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2286","DOI":"10.1109\/TPDS.2014.2347031","article-title":"Goodput-Aware Load Distribution for Real-Time Traffic over Multipath Networks","volume":"26","author":"Wu","year":"2014","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Houze, P., Mory, E., Texier, G., and Simon, G. (2016, January 22\u201327). Applicative-Layer Multipath for Low-Latency Adaptive Live Streaming. Proceedings of the International Conference on Communications (ICC), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ICC.2016.7511550"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.comnet.2020.107638","article-title":"Multipath MMT-based Approach for Streaming High Quality Video over Multiple Wireless Access Networks","volume":"185","author":"Afzal","year":"2021","journal-title":"Comput. Netw."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1109\/TMM.2005.864347","article-title":"MRTP: A Multiflow Real-Time Transport Protocol for Ad Hoc Networks","volume":"8","author":"Mao","year":"2006","journal-title":"IEEE Trans. Multimed."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"66816","DOI":"10.1109\/ACCESS.2021.3076464","article-title":"A Survey on Video Streaming in Multipath and Multihomed Overlay Networks","volume":"9","author":"Hodroj","year":"2021","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Bacco, M., Gotta, A., Roseti, C., and Zampognaro, F. (2014, January 8\u201310). A study on TCP error recovery interaction with Random Access satellite schemes. Proceedings of the 2014 7th Advanced Satellite Multimedia Systems Conference and the 13th Signal Processing for Space Communications Workshop (ASMS\/SPSC), Livorno, Italy.","DOI":"10.1109\/ASMS-SPSC.2014.6934574"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1007\/s00530-013-0319-z","article-title":"A Review of Multiple Description Coding Techniques for Error-Resilient Video Delivery","volume":"20","author":"Kazemi","year":"2014","journal-title":"Multimed. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, Q., Nguyen, T., and Bose, B. (2020, January 17\u201320). Towards Adaptive Packet Scheduler with Deep-Q Reinforcement Learning. Proceedings of the 2020 International Conference on Computing, Networking and Communications (ICNC), Big Island, HI, USA.","DOI":"10.1109\/ICNC47757.2020.9049807"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2295","DOI":"10.1109\/JSAC.2020.3000365","article-title":"Peekaboo: Learning-based multipath scheduling for dynamic heterogeneous environments","volume":"38","author":"Wu","year":"2020","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1109\/TCCN.2019.2952909","article-title":"A deep actor-critic reinforcement learning framework for dynamic multichannel access","volume":"5","author":"Zhong","year":"2019","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/JSYST.2019.2891520","article-title":"An actor-critic deep reinforcement learning approach for transmission scheduling in cognitive internet of things systems","volume":"14","author":"Yang","year":"2019","journal-title":"IEEE Syst. J."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Machumilane, A., Gotta, A., Cassar\u00e1, P., Gennaro, C., and Amato, G. (2022, January 19\u201322). Actor-Critic Scheduling for Path-Aware Air-to-Ground Multipath Multimedia Delivery. Proceedings of the 2022 IEEE 95th Vehicular Technology Conference: (VTC2022-Spring), Helsinki, Finland.","DOI":"10.1109\/VTC2022-Spring54318.2022.9860760"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1109\/MSP.2016.2639062","article-title":"Perfecting Protection for Interactive Multimedia: A survey of forward error correction for low-delay interactive applications","volume":"34","author":"Badr","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_20","first-page":"1","article-title":"Stacking Ensemble Learning for Non-Line-of-Sight Detection of Global Navigation Satellite System","volume":"71","author":"Sun","year":"2022","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_21","first-page":"155","article-title":"Line of sight (los) probability prediction for satellite and haps communication in trabzon, turkey","volume":"1","author":"Hasirci","year":"2016","journal-title":"Int. J. Appl. Math. Electron. Comput."},{"key":"ref_22","unstructured":"Granelli, F. (2020). Computing in Communication Networks, Elsevier."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1109\/4234.996035","article-title":"An analytical model to predict the probability density function of elevation angles for LEO satellite systems","volume":"6","author":"Li","year":"2002","journal-title":"IEEE Commun. Lett."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1109\/25.289418","article-title":"The land mobile satellite communication channel-recording, statistics, and channel model","volume":"40","author":"Lutz","year":"1991","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_25","unstructured":"Bischel, H., Werner, M., and Lutz, E. Proceedings of the Proceedings of Vehicular Technology Conference-VTC, Atlanta, GA, USA, 28 April\u20131 May 1996."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1887","DOI":"10.1109\/TVT.2011.2122253","article-title":"Performance analysis of systematic upper layer FEC codes and interleaving in land mobile satellite channels","volume":"60","author":"Celandroni","year":"2011","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1287\/mnsc.28.1.1","article-title":"State of the art\u2014A survey of partially observable Markov decision processes: Theory, models, and algorithms","volume":"28","author":"Monahan","year":"1982","journal-title":"Manag. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Juan, E., Rodriguez, I., Lauridsen, M., Wigard, J., and Mogensen, P. (2021, January 27\u201330). Time-correlated Geometrical Radio Propagation Model for LEO-to-Ground Satellite Systems. Proceedings of the 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall), Norman, OK, USA.","DOI":"10.1109\/VTC2021-Fall52928.2021.9625273"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1291","DOI":"10.1109\/TSMCC.2012.2218595","article-title":"A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients","volume":"42","author":"Grondman","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.)"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/7\/1842\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:06:59Z","timestamp":1760123219000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/15\/7\/1842"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,30]]},"references-count":29,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["rs15071842"],"URL":"https:\/\/doi.org\/10.3390\/rs15071842","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,30]]}}}