{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T16:59:29Z","timestamp":1794589169416,"version":"4.9.1"},"reference-count":30,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,2,15]],"date-time":"2023-02-15T00:00:00Z","timestamp":1676419200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Adaptation of handover parameters in ultra-dense networks has always been one of the key issues in optimizing network performance. Aiming at the optimization goal of effective handover ratio, this paper proposes a deep Q-learning (DQN) method that dynamically selects handover parameters according to wireless signal fading conditions. This approach seeks good backward compatibility. In order to enhance the efficiency and performance of the DQN method, Long Short Term Memory (LSTM) is used to build a digital twin and assist the DQN algorithm to achieve a more efficient search. Simulation experiments prove that the enhanced method has a faster convergence speed than the ordinary DQN method, and at the same time, achieves an average effective handover ratio increase of 2.7%. Moreover, in different wireless signal fading intervals, the method proposed in this paper has achieved better performance.<\/jats:p>","DOI":"10.3390\/s23042191","type":"journal-article","created":{"date-parts":[[2023,2,16]],"date-time":"2023-02-16T01:36:52Z","timestamp":1676511412000},"page":"2191","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["A Reinforcement Learning Handover Parameter Adaptation Method Based on LSTM-Aided Digital Twin for UDN"],"prefix":"10.3390","volume":"23","author":[{"given":"Jiao","family":"He","sequence":"first","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1242-2409","authenticated-orcid":false,"given":"Tianqi","family":"Xiang","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yixin","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huiyuan","family":"Ruan","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9987-0844","authenticated-orcid":false,"given":"Xin","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Kim, T., Bang, I., and Dan, K.S. (2014, January 2\u20135). Design criteria on a mmWave-based small cell with directional antennas. Proceedings of the IEEE 25th Annual International Symposium on Personal, Indoor, and Mobile Radio Communication, Washington, DC, USA.","DOI":"10.1109\/PIMRC.2014.7136141"},{"key":"ref_2","first-page":"30","article-title":"Ultra Dense Network: Challenges, Enabling Technologies and New Trends","volume":"13","author":"Hao","year":"2016","journal-title":"China Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4557","DOI":"10.1109\/TVT.2021.3070337","article-title":"A Multi-Attribute Handover Algorithm for QoS Enhancement in Ultra Dense Network","volume":"70","author":"Sun","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"4557","DOI":"10.4236\/oalib.1104703","article-title":"Overview of CoMP Clustering in UDN","volume":"5","author":"Li","year":"2018","journal-title":"Oalib"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1682","DOI":"10.1007\/s11036-018-1064-5","article-title":"Energy Efficiency and Superlative TTT for Equitable RLF and Ping Pong in LTE Networks","volume":"23","author":"Kanwal","year":"2018","journal-title":"Mob. Netw."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/MNET.2017.1600301","article-title":"Handover Management in Software-Defined Ultra-Dense 5G Networks","volume":"31","author":"Bilen","year":"2017","journal-title":"IEEE Netw."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wu, M., Huang, W., Sun, K., and Zhang, H. (2020, January 4\u20137). A DQN-Based Handover Management for SDN-Enabled Ultra-Dense Networks. Proceedings of the 2020 IEEE 92nd Vehicular Technology Conference (VTC2020-Fall), Victoria, BC, Canada.","DOI":"10.1109\/VTC2020-Fall49728.2020.9348779"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lu, Y., Zhang, C., Chen, D., Zhang, W., and Xiong, K. (2022, January 18\u201323). Handover Enhancement in High-Speed Railway 5G Networks: A LSTM-based Prediction Method. Proceedings of the 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT), Kharagpur, India.","DOI":"10.1109\/ICCCNT54827.2022.9984602"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1109\/TVT.2018.2874692","article-title":"Frequent-Handover Mitigation in Ultra-Dense Heterogeneous Networks","volume":"68","author":"Hasan","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1895","DOI":"10.1109\/TVT.2013.2247778","article-title":"On the Potential of Handover Parameter Optimization for Self-Organizing Networks","volume":"62","author":"Barco","year":"2013","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kitagawa, K., Komine, T., Yamamoto, T., and Konishi, S. (2011, January 11\u201314). A handover optimization algorithm with mobility robustness for LTE systems. Proceedings of the 2011 IEEE 22nd International Symposium on Personal, Indoor and Mobile Radio Communications, Toronto, ON, Canada.","DOI":"10.1109\/PIMRC.2011.6139784"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"154237","DOI":"10.1109\/ACCESS.2021.3127326","article-title":"A Robust Self-Optimization Algorithm Based on Idiosyncratic Adaptation of Handover Parameters for Mobility Management in LTE-A Heterogeneous Networks","volume":"9","author":"Achhab","year":"2021","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Saad, W., Shayea, I., Hamza, B., Mohamad, H., Daradkeh, Y., and Jabbar, W. (2021). Handover Parameters Optimisation Techniques in 5G Networks. Sensors, 21.","DOI":"10.3390\/s21155202"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Abdelmohsen, A., Abdelwahab, M., Adel, M., Darweesh, M.S., and Mostafa, H. (2018, January 5\u20138). LTE Handover Parameters Optimization Using Q-Learning Technique. Proceedings of the 2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS), Windsor, ON, Canada.","DOI":"10.1109\/MWSCAS.2018.8623826"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1767","DOI":"10.1109\/TITS.2020.3026195","article-title":"Parameter Adaptation and Situation Awareness of LTE-R Handover for High-Speed Railway Communication","volume":"23","author":"Wu","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"214392","DOI":"10.1109\/ACCESS.2020.3037048","article-title":"Individualistic Dynamic Handover Parameter Self-Optimization Algorithm for 5G Networks Based on Automatic Weight Function","volume":"8","author":"Shayea","year":"2020","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"6434","DOI":"10.1109\/TWC.2022.3149415","article-title":"Self-Adapting Handover Parameters Optimization for SDN-Enabled UDN","volume":"21","author":"Huang","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_18","unstructured":"(2023, February 01). Technical Specifification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Resource Control (RRC); Protocol Specifification (R13), 3GPP, Document 36.331 (V13.7.1). Available online: https:\/\/www.3gpp.org\/ftp\/Specs\/archive\/36_series\/36.331."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/MNET.2018.1700235","article-title":"Handover Mechanism in NR for Ultra-Reliable Low-Latency Communications","volume":"32","author":"Park","year":"2018","journal-title":"IEEE Netw."},{"key":"ref_20","unstructured":"Dassanayake, P. (1995, January 3\u20137). Spatial correlation of shadow fading and its impact on handover algorithm parameter settings. Proceedings of the IEEE Singapore International Conference on Networks and International Conference on Information Engineering \u201895, Singapore."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"6060","DOI":"10.1109\/TVT.2016.2640310","article-title":"Analysis of Handover Failures in Heterogeneous Networks With Fading","volume":"66","author":"Vasudeva","year":"2017","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_22","unstructured":"Zheng, H., Rong, C., and Yujiao, C. (2015, January 21\u201323). Theoretical analysis of handover failure rate in heterogenous networks under random shadowed model. Proceedings of the 11th International Conference on Wireless Communications, Networking and Mobile Computing (WiCOM 2015), Shanghai, China."},{"key":"ref_23","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing Atari with Deep Reinforcement Learning. Comput. Sci. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2255","DOI":"10.1109\/COMST.2022.3208773","article-title":"Digital Twins: A Survey on Enabling Technologies, Challenges, Trends and Future Prospects","volume":"24","author":"Mihai","year":"2022","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/MCOM.001.2000343","article-title":"Digital Twin for 5G and Beyond","volume":"59","author":"Nguyen","year":"2021","journal-title":"IEEE Commun. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"112158","DOI":"10.1109\/ACCESS.2022.3215493","article-title":"Empowering 6G Communication Systems With Digital Twin Technology: A Comprehensive Survey","volume":"10","author":"Kuruvatti","year":"2022","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2222","DOI":"10.1109\/TNNLS.2016.2582924","article-title":"LSTM: A Search Space Odyssey","volume":"28","author":"Greff","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Peng, Y., Feng, T., Yang, C., Leng, C., Jiao, L., Zhu, X., Cao, L., and Li, R. (2021, January 13\u201316). HMM-LSTM for Proactive Traffic Prediction in 6G Wireless Networks. Proceedings of the 2021 IEEE 21st International Conference on Communication Technology (ICCT), Tianjin, China.","DOI":"10.1109\/ICCT52962.2021.9657967"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.1109\/LWC.2020.2987791","article-title":"LSTM-Based Active User Number Estimation and Prediction for Cellular Systems","volume":"9","author":"Ding","year":"2020","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/2191\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:36:40Z","timestamp":1760121400000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/2191"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,15]]},"references-count":30,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23042191"],"URL":"https:\/\/doi.org\/10.3390\/s23042191","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,15]]}}}