{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T12:13:33Z","timestamp":1779884013838,"version":"3.53.1"},"reference-count":44,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2021,12,4]],"date-time":"2021-12-04T00:00:00Z","timestamp":1638576000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2021R1G1A1092939"],"award-info":[{"award-number":["2021R1G1A1092939"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Supported by the advances in rocket technology, companies like SpaceX and Amazon competitively have entered the satellite Internet business. These companies said that they could provide Internet service sufficiently to users using their communication resources. However, the Internet service might not be provided in densely populated areas, as the satellites coverage is broad but its resource capacity is limited. To offload the traffic of the densely populated area, we present an adaptable aerial access network (AAN), composed of low-Earth orbit (LEO) satellites and federated reinforcement learning (FRL)-enabled unmanned aerial vehicles (UAVs). Using the proposed system, UAVs could operate with relatively low computation resources than centralized coverage management systems. Furthermore, by utilizing FRL, the system could continuously learn from various environments and perform better with the longer operation times. Based on our proposed design, we implemented FRL, constructed the UAV-aided AAN simulator, and evaluated the proposed system. Base on the evaluation result, we validated that the FRL enabled UAV-aided AAN could operate efficiently in densely populated areas where the satellites cannot provide sufficient Internet services, which improves network performances. In the evaluations, our proposed AAN system provided about 3.25 times more communication resources and had 5.1% lower latency than the satellite-only AAN.<\/jats:p>","DOI":"10.3390\/s21238111","type":"journal-article","created":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T03:10:38Z","timestamp":1638760238000},"page":"8111","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Federated Reinforcement Learning Based AANs with LEO Satellites and UAVs"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0510-535X","authenticated-orcid":false,"given":"Seungho","family":"Yoo","sequence":"first","affiliation":[{"name":"School of Electrical Engineering, Korea University, Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0856-6415","authenticated-orcid":false,"given":"Woonghee","family":"Lee","sequence":"additional","affiliation":[{"name":"Division of IT Convergence Engineering, Hansung University, Seoul 02876, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,4]]},"reference":[{"key":"ref_1","unstructured":"(2021, December 01). Starlink. Available online: https:\/\/www.starlink.com\/."},{"key":"ref_2","unstructured":"(2021, December 01). OneWeb. Available online: https:\/\/oneweb.net\/."},{"key":"ref_3","unstructured":"(2021, December 01). Kuiper Systems, Available online: https:\/\/www.fcc.gov\/document\/fcc-authorizes-kuiper-satellite-constellation."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1109\/MNET.011.2000493","article-title":"Non-Terrestrial Networks in the 6G Era: Challenges and Opportunities","volume":"35","author":"Giordani","year":"2021","journal-title":"IEEE Netw."},{"key":"ref_5","unstructured":"Cartesian (2021, December 01). Starlink RDOF Assessment: Final Report\u2014Prepared for Fiber Broadband Association and NTCA\u2014The Rural Broadband Association, Available online: https:\/\/ecfsapi.fcc.gov\/file\/10208168836021\/FBA_LEO_RDOF_Assessment_Final_Report_20210208.pdf."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.actaastro.2019.03.040","article-title":"A technical comparison of three low earth orbit satellite constellation systems to provide global broadband","volume":"159","author":"Cameron","year":"2019","journal-title":"Acta Astronaut."},{"key":"ref_7","unstructured":"(2021, December 01). LigoWave\u2014Global Wireless Networking Solutions. Available online: https:\/\/www.ligowave.com\/."},{"key":"ref_8","unstructured":"(2021, December 01). Proxim Wireless\u2014Wireless Broadband, Backhaul Solutions and Access Points. Available online: https:\/\/www.proxim.com\/."},{"key":"ref_9","unstructured":"(2021, December 01). FiberLight: Building Better Fiber Networks. Available online: https:\/\/www.fiberlight.com\/."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.icte.2019.12.002","article-title":"FSO as backhaul and energizer for drone-assisted mobile access networks","volume":"6","author":"Ansari","year":"2020","journal-title":"ICT Express"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1016\/j.icte.2019.08.003","article-title":"Flexible backhaul-aware DBS-aided HetNet with IBFD communications","volume":"6","author":"Ansari","year":"2020","journal-title":"ICT Express"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"18391","DOI":"10.1109\/ACCESS.2017.2735988","article-title":"LEO satellite constellation for Internet of Things","volume":"5","author":"Qu","year":"2017","journal-title":"IEEE Access"},{"key":"ref_13","unstructured":"(2021, December 01). High Altitude Connectivity: The Next Chapter. Available online: https:\/\/engineering.fb.com\/2018\/06\/27\/connectivity\/high-altitude-connectivity-the-next-chapter\/."},{"key":"ref_14","unstructured":"(2021, December 01). Zephyr\u2014Airbus. Available online: https:\/\/www.airbus.com\/en\/products-services\/defence\/uas\/uas-solutions\/zephyr."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1109\/MWC.2017.1600173","article-title":"Toward a Flexible and Reconfigurable Broadband Satellite Network: Resource Management Architecture and Strategies","volume":"24","author":"Sheng","year":"2017","journal-title":"IEEE Wirel. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1002\/sat.1169","article-title":"Joint cooperative spectrum sensing and channel selection optimization for satellite communication systems based on cognitive radio","volume":"35","author":"Jia","year":"2017","journal-title":"Int. J. Satell. Commun. Netw."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Sharma, S.K., Chatzinotas, S., and Ottersten, B. (2013, January 2\u20135). Cognitive Radio Techniques for Satellite Communication Systems. Proceedings of the 2013 IEEE 78th Vehicular Technology Conference (VTC Fall), Las Vegas, NV, USA.","DOI":"10.1109\/VTCFall.2013.6692139"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1109\/TNET.2002.1012371","article-title":"MLSR: A novel routing algorithm for multilayered satellite IP networks","volume":"10","author":"Akyildiz","year":"2002","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"5550","DOI":"10.1109\/ACCESS.2019.2963223","article-title":"Convergence of satellite and terrestrial networks: A comprehensive survey","volume":"8","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_20","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/j.icte.2020.06.002","article-title":"Edge computational task offloading scheme using reinforcement learning for IIoT scenario","volume":"6","author":"Hossain","year":"2020","journal-title":"ICT Express"},{"key":"ref_22","first-page":"1","article-title":"Exploring strategies for training deep neural networks","volume":"10","author":"Larochelle","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_23","unstructured":"Fan, J., Wang, Z., Xie, Y., and Yang, Z. (2020, January 11\u201312). A theoretical analysis of deep Q-learning. Proceedings of the Learning for Dynamics and Control, Virtual Meeting."},{"key":"ref_24","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_25","unstructured":"Babaeizadeh, M., Frosio, I., Tyree, S., Clemons, J., and Kautz, J. (2016). Reinforcement learning through asynchronous advantage actor-critic on a gpu. arXiv."},{"key":"ref_26","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, January 7\u20139). Trust region policy optimization. Proceedings of the International Conference on Machine Learning, Lille, France."},{"key":"ref_27","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_28","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_29","unstructured":"Kone\u010dn\u1ef3, J., McMahan, H.B., Ramage, D., and Richt\u00e1rik, P. (2016). Federated optimization: Distributed machine learning for on-device intelligence. arXiv."},{"key":"ref_30","unstructured":"Mammen, P.M. (2021). Federated Learning: Opportunities and Challenges. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"53841","DOI":"10.1109\/ACCESS.2020.2981430","article-title":"Federated learning for UAVs-enabled wireless networks: Use cases, challenges, and open problems","volume":"8","author":"Brik","year":"2020","journal-title":"IEEE Access"},{"key":"ref_32","unstructured":"Zhuo, H.H., Feng, W., Xu, Q., Yang, Q., and Lin, Y. (2019). Federated reinforcement learning. arXiv."},{"key":"ref_33","first-page":"2059","article-title":"Reinforcement learning-based control of nonlinear systems using Lyapunov stability concept and fuzzy reward scheme","volume":"67","author":"Chen","year":"2019","journal-title":"IEEE Trans. Circuits Syst. II Express Briefs"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Azar, A.T., Koubaa, A., Ali Mohamed, N., Ibrahim, H.A., Ibrahim, Z.F., Kazim, M., Ammar, A., Benjdira, B., Khamis, A.M., and Hameed, I.A. (2021). Drone Deep Reinforcement Learning: A Review. Electronics, 10.","DOI":"10.3390\/electronics10090999"},{"key":"ref_35","first-page":"8026","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume":"32","author":"Paszke","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","unstructured":"Tabor, P. (2021, December 01). Youtube-Code-Repository. Available online: https:\/\/github.com\/philtabor\/Youtube-Code-Repository\/tree\/master\/ReinforcementLearning\/PolicyGradient\/PPO\/torch."},{"key":"ref_37","unstructured":"Jadhav, A.R. (2021, December 01). Federated-Learning (PyTorch). Available online: https:\/\/github.com\/AshwinRJ\/Federated-Learning-PyTorch."},{"key":"ref_38","unstructured":"Sakai, A., Ingram, D., Dinius, J., Chawla, K., Raffin, A., and Paques, A. (2018). PythonRobotics: A Python code collection of robotics algorithms. arXiv."},{"key":"ref_39","unstructured":"(2021, December 01). Evaluation YouTube Video. Available online: https:\/\/youtu.be\/I3T3ehmJeck."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Riley, G.F., and Henderson, T.R. (2010). The ns-3 network simulator. Modeling and Tools for Network Simulation, Springer.","DOI":"10.1007\/978-3-642-12331-3_2"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kassing, S., Bhattacherjee, D., \u00c1guas, A.B., Saethre, J.E., and Singla, A. (2020, January 27\u201329). Exploring the \u201cInternet from Space\u201d with Hypatia. Proceedings of the ACM Internet Measurement Conference (IMC \u201920), Virtual Meeting.","DOI":"10.1145\/3419394.3423635"},{"key":"ref_42","unstructured":"(2021, December 01). Ericsson Mobility Report, June 2021. Available online: https:\/\/www.ericsson.com\/en\/reports-and-papers\/mobility-report\/reports\/june-2021."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1109\/TBDATA.2017.2778721","article-title":"Understanding Urban Dynamics From Massive Mobile Traffic Data","volume":"5","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Big Data"},{"key":"ref_44","unstructured":"CIESIN (2010). Gridded population of the world version 4 (GPWV4): Population density grids. Socioeconomic Data and Applications Center (SEDAC), Columbia University."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/8111\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:39:40Z","timestamp":1760168380000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/8111"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,4]]},"references-count":44,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21238111"],"URL":"https:\/\/doi.org\/10.3390\/s21238111","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,4]]}}}