{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T15:35:28Z","timestamp":1775230528063,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JST SPRING","award":["PMJSP2123"],"award-info":[{"award-number":["PMJSP2123"]}]},{"name":"JST SPRING","award":["05701"],"award-info":[{"award-number":["05701"]}]},{"name":"National Institute of Information and Communications Technology (NICT), Japan","award":["PMJSP2123"],"award-info":[{"award-number":["PMJSP2123"]}]},{"name":"National Institute of Information and Communications Technology (NICT), Japan","award":["05701"],"award-info":[{"award-number":["05701"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>In this paper, we propose a novel Deep Reinforcement Learning Evolution Algorithm (DRLEA) method to control the antenna parameters of the High-Altitude Platform Station (HAPS) mobile to reduce the number of low-throughput users. Considering the random movement of the HAPS caused by the winds, the throughput of the users might decrease. Therefore, we propose a method that can dynamically adjust the antenna parameters based on the throughput of the users in the coverage area to reduce the number of low-throughput users by improving the users\u2019 throughput. Different from other model-based reinforcement learning methods, such as the Deep Q Network (DQN), the proposed method combines the Evolution Algorithm (EA) with Reinforcement Learning (RL) to avoid the sub-optimal solutions in each state. Moreover, we consider non-uniform user distribution scenarios, which are common in the real world, rather than ideal uniform user distribution scenarios. To evaluate the proposed method, we do the simulations under four different real user distribution scenarios and compare the proposed method with the conventional EA and RL methods. The simulation results show that the proposed method effectively reduces the number of low throughput users after the HAPS moves.<\/jats:p>","DOI":"10.3390\/fi15010034","type":"journal-article","created":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T03:47:01Z","timestamp":1673495221000},"page":"34","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Deep Reinforcement Learning Evolution Algorithm for Dynamic Antenna Control in Multi-Cell Configuration HAPS System"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9483-1419","authenticated-orcid":false,"given":"Siyuan","family":"Yang","sequence":"first","affiliation":[{"name":"Graduate School of Science and Technology, Keio University, Yokohama 223-8522, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7055-9318","authenticated-orcid":false,"given":"Mondher","family":"Bouazizi","sequence":"additional","affiliation":[{"name":"Department of Information and Computer Science, Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3961-1426","authenticated-orcid":false,"given":"Tomoaki","family":"Ohtsuki","sequence":"additional","affiliation":[{"name":"Department of Information and Computer Science, Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9037-5433","authenticated-orcid":false,"given":"Yohei","family":"Shibata","sequence":"additional","affiliation":[{"name":"SoftBank Corp. Technology Research Laboratory, Tokyo 135-0064, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wataru","family":"Takabatake","sequence":"additional","affiliation":[{"name":"SoftBank Corp. Technology Research Laboratory, Tokyo 135-0064, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kenji","family":"Hoshino","sequence":"additional","affiliation":[{"name":"SoftBank Corp. Technology Research Laboratory, Tokyo 135-0064, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Atsushi","family":"Nagate","sequence":"additional","affiliation":[{"name":"SoftBank Corp. Technology Research Laboratory, Tokyo 135-0064, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1049\/ecej:20010303","article-title":"High-Altitude Platforms for Wireless Communications","volume":"13","author":"Tozer","year":"2001","journal-title":"Electron. Commun. Eng. J. (ECEJ)"},{"key":"ref_2","unstructured":"(2019, August 26). Why SoftBank Is Looking to the Stratosphere. Softbank. Available online: https:\/\/www.softbank.jp\/en\/sbnews\/entry\/20190826_01."},{"key":"ref_3","unstructured":"Tereza, P., and Elizabeth, H. (2022, November 23). Starlink Satellites: Everything You Need to Know about the Controversial Internet Megaconstellation. SPACE. Available online: https:\/\/www.space.com\/spacex-starlink-satellites.html."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2096","DOI":"10.1109\/LWC.2021.3091164","article-title":"Aerial RIS-Assisted High Altitude Platform Communications","volume":"10","author":"Gao","year":"2021","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1109\/COMST.2005.1423332","article-title":"Broadband communications via high-altitude platforms: A survey","volume":"7","author":"Karapantazis","year":"2005","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1362","DOI":"10.1109\/TWC.2021.3103764","article-title":"Joint Optimization of Transmission and Computation Resources for Satellite and High Altitude Platform Assisted Edge Computing","volume":"21","author":"Ding","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"7897","DOI":"10.1109\/TWC.2020.3017170","article-title":"Space-Air-Ground Integrated Networks: Outage Performance Analysis","volume":"19","author":"Ye","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"6478","DOI":"10.1109\/TWC.2022.3149903","article-title":"An Energy-efficient Aerial Backhaul System with Reconfigurable Intelligent Surface","volume":"21","author":"Jeon","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"58","DOI":"10.23919\/ICN.2020.0004","article-title":"Overview of development and regulatory aspects of high altitude platform system","volume":"1","author":"Zhou","year":"2020","journal-title":"Intell. Converg. Networks"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1007\/s11277-006-9184-9","article-title":"High Altitude Platform Station (HAPS): A Review of New Infrastructure Development for Future Wireless Communications","volume":"42","author":"Widiawan","year":"2007","journal-title":"Wirel. Pers. Commun."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1939","DOI":"10.1109\/JPROC.2011.2159690","article-title":"The Role of High-Altitude Platforms (HAPs) in the Global Wireless Connectivity","volume":"99","author":"Mohammed","year":"2011","journal-title":"Proc. IEEE"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2253","DOI":"10.1109\/TVT.2007.897185","article-title":"Data Communications to Trains From High-Altitude Platforms","volume":"56","author":"White","year":"2007","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_13","unstructured":"El-Jabu, B., and Steele, R. (1999, January 19\u201322). Effect of positional instability of an aerial platform on its CDMA performance. Proceedings of the Gateway to 21st Century Communications Village, VTC 1999-Fall, IEEE VTS 50th Vehicular Technology Conference (Cat. No.99CH36324), Amsterdam, The Netherlands."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1483","DOI":"10.1109\/TWC.2005.850282","article-title":"Effect of Lateral Displacement of a High-Altitude Platform on Cellular Interference and Handover","volume":"4","author":"Thornton","year":"2005","journal-title":"Trans. Wireless. Commun."},{"key":"ref_15","unstructured":"Panfeng, H., Naiping, C., and Shuyan, N. (2016, January 26\u201328). Coverage model of multi beam antenna from high altitude platform in the swing state. Proceedings of the 2016 7th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"263","DOI":"10.2528\/PIER06092502","article-title":"Geometrical Analysis of High Altitude Platforms Cellular Footprint","volume":"67","author":"Dessouky","year":"2007","journal-title":"Prog. Electromagn. Res.-Pier"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dessouky, M., Nofal, M., Sharshar, H., and Albagory, Y. (2006, January 14\u201316). Optimization of Beams Directions for High Altitude Platforms Cellular Communications Design. Proceedings of the Twenty Third National Radio Science Conference (NRSC\u20192006), Menouf, Egypt.","DOI":"10.1109\/NRSC.2006.386352"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2651","DOI":"10.1007\/s11277-015-2758-7","article-title":"Handover Performance of Unstable-Yaw Stratospheric High-Altitude Stations","volume":"84","author":"Albagory","year":"2015","journal-title":"Wirel. Pers. Commun."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"He, P., Cheng, N., and Cui, J. (2016, January 13\u201315). Handover performance analysis of cellular communication system from high altitude platform in the swing state. Proceedings of the 2016 IEEE International Conference on Signal and Image Processing (ICSIP), Beijing, China.","DOI":"10.1109\/SIPROCESS.2016.7888294"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hoshino, K., Sudo, S., and Ohta, Y. (2019, January 22\u201325). A Study on Antenna Beamforming Method Considering Movement of Solar Plane in HAPS System. Proceedings of the 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.","DOI":"10.1109\/VTCFall.2019.8891546"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Enache, F., Dep\u0103r\u0103\u0163eanu, D., and Popescu, F. (July, January 29). Optimal design of circular antenna array using genetic algorithms. Proceedings of the 2017 9th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), Targoviste, Romania.","DOI":"10.1109\/ECAI.2017.8166392"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Sun, G., Liu, Y., Chen, Z., Zhang, Y., Wang, A., and Liang, S. (2017, January 19\u201322). Thinning of Concentric Circular Antenna Arrays Using Improved Discrete Cuckoo Search Algorithm. Proceedings of the 2017 IEEE Wireless Communications and Networking Conference (WCNC), San Francisco, CA, USA.","DOI":"10.1109\/WCNC.2017.7925959"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3859","DOI":"10.1007\/s00521-017-2971-2","article-title":"Design of planar concentric circular antenna arrays with reduced side lobe level using symbiotic organisms search","volume":"30","author":"Dib","year":"2018","journal-title":"Neural Comput. Appl."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.aeue.2018.05.002","article-title":"Performance Improvement of High Altitude Platform Using Concentric Circular Antenna Array Based on Particle Swarm Optimization","volume":"91","author":"Ismaiel","year":"2018","journal-title":"AEU\u2014Int. J. Electron. Commun."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"848","DOI":"10.1016\/j.protcy.2012.10.103","article-title":"Thinned Concentric Circular Antenna Array Synthesis using Particle Swarm Optimization","volume":"6","author":"Mandal","year":"2012","journal-title":"Procedia Technol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Arum, S.C., Grace, D., Mitchell, P.D., and Zakaria, M.D. (2019, January 22\u201325). Beam-Pointing Algorithm for Contiguous High-Altitude Platform Cell Formation for Extended Coverage. Proceedings of the 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.","DOI":"10.1109\/VTCFall.2019.8891528"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1109\/TWC.2004.842972","article-title":"Improving the system capacity of broadband services using multiple high-altitude platforms","volume":"4","author":"Grace","year":"2005","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1109\/TWC.2003.811052","article-title":"Optimizing an array of antennas for cellular coverage from a high altitude platform","volume":"2","author":"Thornton","year":"2003","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"101896","DOI":"10.1016\/j.phycom.2022.101896","article-title":"Deep reinforcement learning based IRS-assisted mobile edge computing under physical-layer security","volume":"55","author":"Zhang","year":"2022","journal-title":"Phys. Commun."},{"key":"ref_30","first-page":"2730","article-title":"Offline Reinforcement Learning with Realizability and Single-policy Concentrability","volume":"Volume 178","author":"Loh","year":"2022","journal-title":"Proceedings of the Thirty Fifth Conference on Learning Theory; Proceedings of Machine Learning Research, London, UK, 2\u20135 July 2022"},{"key":"ref_31","unstructured":"Lazaridou, A., and Baroni, M. (2020). Emergent Multi-Agent Communication in the Deep Learning Era. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1581","DOI":"10.1109\/TCOMM.2019.2961332","article-title":"Deep Reinforcement Learning for 5G Networks: Joint Beamforming, Power Control, and Interference Coordination","volume":"68","author":"Mismar","year":"2020","journal-title":"IEEE Trans. Commun."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Rkhami, A., Hadjadj-Aoul, Y., and Outtagarts, A. (2021, January 9\u201312). Learn to improve: A novel deep reinforcement learning approach for beyond 5G network slicing. Proceedings of the 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA.","DOI":"10.1109\/CCNC49032.2021.9369463"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2454","DOI":"10.1049\/cmu2.12501","article-title":"Deep learning integrated reinforcement learning for adaptive beamforming in B5G networks","volume":"16","author":"Eappen","year":"2022","journal-title":"IET Commun."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wada, K., Yang, S., Bouazizi, M., Ohtsuki, T., Shibata, Y., Takabatake, W., Hoshino, K., and Nagate, A. (2022, January 16\u201320). Dynamic Antenna Control for HAPS Using Fuzzy Q-Learning in Multi-Cell Configuration. Proceedings of the ICC 2022\u2014IEEE International Conference on Communications, Seoul, Republic of Korea.","DOI":"10.1109\/ICC45855.2022.9838271"},{"key":"ref_36","unstructured":"(2000). Minimum Performance Characteristics and Operational Conditions for High Altitude Platform Stations Providing IMT-2000 in the Bands 1885\u20131980 MHz, 2010\u20132025 MHz and 2110\u20132170 MHz in Regions 1 and 3 and 1885\u20131980 MHz and 2110\u20132160 MHz in Region 2, Document ITU-R M.1456, International Telecommunications Union."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"157995","DOI":"10.1109\/ACCESS.2020.3019820","article-title":"System Design of Gigabit HAPS Mobile Communications","volume":"8","author":"Shibata","year":"2020","journal-title":"IEEE Access"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"133653","DOI":"10.1109\/ACCESS.2019.2941229","article-title":"Q-Learning Algorithms: A Comprehensive Classification and Applications","volume":"7","author":"Jang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_40","unstructured":"Hasselt, H.v., Guez, A., and Silver, D. (2016, January 12\u201317). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI\u201916, Phoenix, AZ, USA."},{"key":"ref_41","unstructured":"(2021, December 06). Dynamic Population Data. Agoop Corporation. Available online: https:\/\/www.agoop.co.jp\/service\/dynamic-population-data\/."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/1\/34\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:04:01Z","timestamp":1760119441000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/1\/34"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,12]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["fi15010034"],"URL":"https:\/\/doi.org\/10.3390\/fi15010034","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,12]]}}}