{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T21:47:10Z","timestamp":1783460830150,"version":"3.55.0"},"reference-count":50,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,1,27]],"date-time":"2024-01-27T00:00:00Z","timestamp":1706313600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea (NRF)","doi-asserted-by":"publisher","award":["2022R1F1A1062696"],"award-info":[{"award-number":["2022R1F1A1062696"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The directional antenna combined with beamforming is one of the attractive solutions to accommodate high data rate applications in 5G vehicle communications. However, the directional nature of beamforming requires beam alignment between the transmitter and the receiver, which incurs significant signaling overhead. Hence, we need to find the optimal parameters for directional beamforming, i.e., the antenna beamwidth and beam alignment interval, that maximize the throughput, taking the beam alignment overhead into consideration. In this paper, we propose a reinforcement learning (RL)-based beamforming scheme in a vehicle-to-infrastructure system, where we jointly determine the antenna beamwidth and the beam alignment interval, taking into account the past and future rewards. The simulation results show that the proposed RL-based joint beamforming scheme outperforms conventional beamforming schemes in terms of the average throughput and the average link stability ratio.<\/jats:p>","DOI":"10.3390\/s24030837","type":"journal-article","created":{"date-parts":[[2024,1,30]],"date-time":"2024-01-30T12:06:58Z","timestamp":1706616418000},"page":"837","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Reinforcement Learning-Based Joint Beamwidth and Beam Alignment Interval Optimization in V2I Communications"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-5414-1019","authenticated-orcid":false,"given":"Jihun","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6015-5041","authenticated-orcid":false,"given":"Hun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7868-7197","authenticated-orcid":false,"given":"Jaewoo","family":"So","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5009","DOI":"10.1109\/TITS.2021.3101259","article-title":"Guest editorial special issue on intent-based networking for 5G-envisioned Internet of connected vehicles","volume":"22","author":"Garg","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1972","DOI":"10.1109\/COMST.2021.3057017","article-title":"A tutorial on 5G NR V2X communications","volume":"23","author":"Garcia","year":"2021","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Han, D., and So, J. (2023). Energy-efficient resource allocation based on deep Q-network in V2V communications. Sensors, 23.","DOI":"10.3390\/s23031295"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1571","DOI":"10.1109\/COMST.2023.3293231","article-title":"A Survey on applications of cache-aided NOMA","volume":"25","author":"Bepari","year":"2023","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"13340","DOI":"10.1109\/TVT.2021.3122257","article-title":"A hybrid deep reinforcement learning for autonomous vehicles smart-platooning","volume":"70","author":"Prathiba","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, Z., Sung, G., Su, H., Yu, H., Lei, B., and Guizani, M. (2023). Low-latency scheduling approach for dependent tasks in MEC-enabled 5G vehicular networks. IEEE Internet Things J., 1\u201312.","DOI":"10.1109\/JIOT.2023.3309940"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Ju, Y., Cao, Z., Chen, Y., Liu, L., Pei, Q., Mumtaz, S., Dong, M., and Guizani, M. (2023). NOMA-assisted secure offloading for vehicular edge computing networks with asynchronous deep reinforcement learning. IEEE Trans. Intell. Transp. Syst., 1\u201314.","DOI":"10.1109\/TITS.2023.3320861"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shahzad, M., and Antoniou, J. (2019, January 11\u201313). Quality of user experience in 5G-VANET. Proceedings of the 2019 IEEE 24th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), Limassol, Cyprus.","DOI":"10.1109\/CAMAD.2019.8858442"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"105631","DOI":"10.1109\/ACCESS.2022.3210985","article-title":"Steet smart in 5G: Vehicular applications, communication, and computing","volume":"10","author":"Alhilal","year":"2022","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Raj, T., Mishra, R., Kumar, P., and Kapoor, A. (2023). Advances in MIMO antenna design for 5G: A comprehensive review. Sensors, 23.","DOI":"10.3390\/s23146329"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e07931","DOI":"10.1016\/j.heliyon.2021.e07931","article-title":"Wideband miniaturized patch radiator for sub-6 GHz 5G devices","volume":"7","author":"Kapoor","year":"2021","journal-title":"Heliyon"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"27","DOI":"10.7716\/aem.v10i2.1687","article-title":"A compact high gain printed antenna with frequency selective surface for 5G wideband applications","volume":"10","author":"Kapoor","year":"2021","journal-title":"Adv. Electromagn."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e09336","DOI":"10.1016\/j.heliyon.2022.e09336","article-title":"High gain modified Vivaldi vehicular antenna for IoV communications in 5G network","volume":"8","author":"Kapoor","year":"2022","journal-title":"Heliyon"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1109\/MCOM.2016.1600071CM","article-title":"Millimeter-wave vehicular communication to support massive automotive sensing","volume":"54","author":"Choi","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7547","DOI":"10.1109\/TCOMM.2020.3026377","article-title":"Bayesian beamforming for mobile millimeter wave channel tracking in the presence of DOA uncertainty","volume":"68","author":"Yang","year":"2020","journal-title":"IEEE Trans. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"949","DOI":"10.1109\/COMST.2015.2504600","article-title":"Beamforming for millimeter wave communications: An inclusive survey","volume":"18","author":"Kutty","year":"2016","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Shokri-Ghadikolaei, H., Gkatzikis, L., and Fischione, C. (2015, January 8\u201312). Beam-searching and transmission scheduling in millimeter wave communications. Proceedings of the 2015 IEEE International Conference on Communications (ICC), London, UK.","DOI":"10.1109\/ICC.2015.7248501"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1654","DOI":"10.1109\/COMST.2018.2816920","article-title":"IEEE 802.11ay-based mmWave WLANs: Design challenges and solutions","volume":"20","author":"Zhou","year":"2018","journal-title":"IEEE Commun. Surv. Tuts."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Tang, Z., Wang, J., Wang, J., and Song, J. (2018, January 9\u201313). A high-accuracy adaptive beam training algorithm for mmWave communication. Proceedings of the 2018 IEEE Globecom Workshops (GC Wkshps), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/GLOCOMW.2018.8644165"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4792","DOI":"10.1109\/TWC.2018.2831697","article-title":"A scalable and statistically robust beam alignment technique for millimeter-wave systems","volume":"17","author":"Song","year":"2018","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1109\/TVT.2021.3049380","article-title":"A deep learning-based low overhead beam selection in mmWave communications","volume":"70","author":"Echigo","year":"2021","journal-title":"IEEE Trans. Veh. Tech."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"6481","DOI":"10.1109\/TWC.2021.3074700","article-title":"User scheduling and beam alignment in mmWave networks with a large number of mobile users","volume":"20","author":"Lee","year":"2021","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Brambilla, M., Combi, L., Matera, A., Tagliaferri, D., Nicoli, M., and Spagnolini, U. (2020). Sensor-aided V2X beam tracking for connected automated driving: Distributed architecture and processing algorithms. Sensors, 20.","DOI":"10.3390\/s20123573"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"19622","DOI":"10.3390\/s141019622","article-title":"Application of motion sensors for beam-tracking of mobile stations in mmWave communication systems","volume":"14","author":"Shim","year":"2014","journal-title":"Sennsors"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Wang, J., Feng, Y., He, D., Xu, Y., Zheng, H., and Zhang, W. (2019, January 22\u201325). Beam design for beam training based millimeter wave V2I communications. Proceedings of the IEEE Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.","DOI":"10.1109\/VTCFall.2019.8891230"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"110601","DOI":"10.1109\/ACCESS.2021.3102230","article-title":"Antenna beamwidth optimization in directional device-to-device communication using multi-agent deep reinforcement learning","volume":"9","author":"Bahadori","year":"2021","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2442","DOI":"10.1109\/TWC.2020.3042518","article-title":"Joint beamwidth and power optimization in mmWave hybrid beamforming NOMA systems","volume":"20","author":"Almasi","year":"2021","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Coll-Perales, B., Gozalvez, J., and Egea-Lopez, E. (2021, January 10\u201312). Adaptive beamwidth configuration for millimeter wave V2X scheduling. Proceedings of the 2021 IEEE Vehicular Networking Conference (VNC), Ulm, Germany.","DOI":"10.1109\/VNC52810.2021.9644656"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2201","DOI":"10.1109\/LCOMM.2020.3005138","article-title":"Deep reinforcement learning for joint beamwidth and power optimization in mmWave systems","volume":"24","author":"Gao","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Feng, Y., Wang, J., He, D., and Guan, Y. (2019, January 22\u201325). Beam design for V2V communications with inaccurate positioning based on millimeter wave. Proceedings of the 2019 IEEE Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.","DOI":"10.1109\/VTCFall.2019.8891564"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hao, W., Zhou, F., Chu, Z., Xiao, P., Tafazolli, R., and Al-Dhahir, N. (2019, January 20\u201324). Beam alignment for MIMO-NOMA millimeter wave communication systems. Proceedings of the 2019 IEEE International Conference on Communications (ICC), Shanghai, China.","DOI":"10.1109\/ICC.2019.8762058"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Jamil, S., Rahman, M., Tanveer, J., and Haider, A. (2022). Energy efficiency and throughput maximization using millimeter waves-microwaves HetNets. Electronics, 11.","DOI":"10.3390\/electronics11030474"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"6806","DOI":"10.1109\/TWC.2018.2864650","article-title":"An efficient uplink multi-connectivity scheme for 5G millimeter-wave control plane applications","volume":"17","author":"Giordani","year":"2018","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1109\/TVT.2020.3044926","article-title":"Joint transmission reception point selection and resource allocation for energy-efficient millimeter-wave communications","volume":"70","author":"Liu","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"5052","DOI":"10.1109\/TAP.2018.2855668","article-title":"A reconfigurable antenna with beam steering and beamwidth variability for wireless communications","volume":"66","author":"Towfiq","year":"2018","journal-title":"IEEE Trans. Antennas Propag."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"512","DOI":"10.1109\/JCN.2014.000090","article-title":"Fast millimeter-wave beam training with receive beamforming","volume":"16","author":"Kim","year":"2014","journal-title":"J. Commun. Netw."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1109\/TCOMM.2017.2767041","article-title":"Performance analysis of millimeter-wave relaying: Impacts of beamwidth and self-interference","volume":"66","author":"Yang","year":"2018","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_38","first-page":"1619","article-title":"Millimeter wave beam alignment: Large deviations analysis and design insights","volume":"35","author":"Liu","year":"2017","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Van Trees, H.L. (2002). Optimum Array Processing: Part IV of Detection, Estimation, and Modulation Theory, Wiley.","DOI":"10.1002\/0471221104"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1109\/JSTSP.2019.2901593","article-title":"NOMA for hybrid mmWave communication systems with beamwidth control","volume":"13","author":"Wei","year":"2019","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2473","DOI":"10.1109\/TMTT.2022.3151680","article-title":"Analysis of ultralow power radio frequency beamforming using transmission-line transformers and tunable passives","volume":"70","author":"Anderson","year":"2022","journal-title":"IEEE Trans. Microw. Theory Tech."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"26","DOI":"10.4218\/etrij.2017-0188","article-title":"Field-measurement-based received power analysis for directional beamforming millimeter-wave systems: Effects of beamwidth and beam misalignment","volume":"40","author":"Lee","year":"2018","journal-title":"ETRI J."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Va, V., Shimizu, T., Bansal, G., and Heath, R.W. (2016, January 22\u201327). Beam design for beam switching based millimeter wave vehicle-to infrastructure communications. Proceedings of the 2016 IEEE International Conference on Communications (ICC), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ICC.2016.7511414"},{"key":"ref_44","unstructured":"Heino, P., Meinil\u00e4, J., Ky\u00f6sti, P., Hentila, L., J\u00e4ms\u00e4, T., Suikkanen, E., Kunnari, E., and Narandzic, M. (2010). D5.3: WINNER+ final channel models. CP5-026 WINNER+, Version 1.0."},{"key":"ref_45","unstructured":"(2016). 3GPP. Study on LTE-Based V2X Services. 3GPP TR 36.885 V14.0.0."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1755","DOI":"10.1109\/TWC.2017.2785249","article-title":"A WINNER+ based 3-D non-stationary wideband MIMO channel model","volume":"17","author":"Bian","year":"2018","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2001","DOI":"10.1109\/JIOT.2021.3089165","article-title":"A novel fairness allocation strategy with minimum mainlobe interference for mmWave networks","volume":"9","author":"Yu","year":"2022","journal-title":"IEEE Internet Things J."},{"key":"ref_48","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Maeng, S.J., Park, S.H., Moon, S.H., and Cho, Y.S. (2018, January 27\u201330). Inter-beam interference reduction technique for millimeter-wave cellular systems using hybrid beamforming. Proceedings of the 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall), Chicago, IL, USA.","DOI":"10.1109\/VTCFall.2018.8690999"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1016\/j.dcan.2018.01.003","article-title":"Beam interference suppression in multi-cell millimeter wave communications","volume":"5","author":"Zhang","year":"2019","journal-title":"Digit. Commun. Netw."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/3\/837\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:50:23Z","timestamp":1760104223000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/3\/837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,27]]},"references-count":50,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,2]]}},"alternative-id":["s24030837"],"URL":"https:\/\/doi.org\/10.3390\/s24030837","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,27]]}}}