{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T14:25:06Z","timestamp":1779373506808,"version":"3.53.1"},"reference-count":40,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T00:00:00Z","timestamp":1620172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2019R1A2C1008530"],"award-info":[{"award-number":["2019R1A2C1008530"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>5G and Beyond 5G mobile networks use several high-frequency spectrum bands such as the millimeter-wave (mmWave) bands to alleviate the problem of bandwidth scarcity. However high-frequency bands do not cover larger distances. The coverage problem is addressed by using a heterogeneous network which comprises numerous small and macrocells, defined by transmission and reception points (TRxPs). For such a network, random access is considered a challenging function in which users attempt to select an efficient TRxP by random access within a given time. Ideally, an efficient TRxP is less congested, minimizing delays in users\u2019 random access. However, owing to the nature of random access, it is not feasible to deploy a centralized controller estimating the congestion level of each cell and deliver this information back to users during random access. To solve this problem, we establish an optimization problem and employ a reinforcement-learning-based scheme. The proposed scheme estimates congestion of TRxPs in service and selects the optimal access point. Mathematically, this approach is beneficial in approximating and minimizing a random access delay function. Through simulation, we demonstrate that our proposed deep learning-based algorithm improves performance on random access. Notably, the average access delay is improved by 58.89% from the original 3GPP algorithm, and the probability of successful access also improved.<\/jats:p>","DOI":"10.3390\/s21093210","type":"journal-article","created":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T22:51:42Z","timestamp":1620255102000},"page":"3210","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Random Access Using Deep Reinforcement Learning in Dense Mobile Networks"],"prefix":"10.3390","volume":"21","author":[{"given":"Yared Zerihun","family":"Bekele","sequence":"first","affiliation":[{"name":"Department of Artificial Intelligence, Ajou University, Suwon 16499, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Young-June","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Software and Computer Engineering, Ajou University, Suwon 16499, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,5]]},"reference":[{"key":"ref_1","unstructured":"(2021, March 10). 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Resource Control (RRC); Protocol Specification. Available online: https:\/\/www.3gpp.org\/ftp\/Specs\/archive\/36_series\/36.331."},{"key":"ref_2","unstructured":"(2021, March 10). 3rd Generation Partnership Project; Radio Resource Control (RRC); Protocol Specification. Available online: https:\/\/www.3gpp.org\/ftp\/Specs\/archive\/38_series\/38.331."},{"key":"ref_3","unstructured":"(2021, March 10). 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA) User Equipment (UE) Procedures in idle Mode. Available online: https:\/\/www.3gpp.org\/ftp\/Specs\/archive\/36_series\/36.304."},{"key":"ref_4","unstructured":"(2021, March 10). 3rd Generation Partnership Project; Technical Specification Group Core Network; NAS Functions related to Mobile Station (MS) in idle Mode. Available online: https:\/\/www.3gpp.org\/ftp\/Specs\/archive\/23_series\/23.122."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wan, C., and Sun, J. (2019, January 5\u20137). Access Class Barring Parameter Adaptation Based on Load Estimation Model for mMTC in LTE-A. Proceedings of the 2019 International Conference on Communications, Information System and Computer Engineering (CISCE), Haikou, China.","DOI":"10.1109\/CISCE.2019.00119"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tello-Oquendo, L., Vidal, J.R., Pla, V., and Guijarro, L. (2018, January 20\u201322). Dynamic access class barring parameter tuning in LTE-A networks with massive M2M traffic. Proceedings of the 2018 17th Annual Mediterranean Ad Hoc Networking Workshop (Med-Hoc-Net), Capri, Italy.","DOI":"10.23919\/MedHocNet.2018.8407086"},{"key":"ref_7","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","author":"Kaelbling","year":"1996","journal-title":"J. Artif. Intell. Res."},{"key":"ref_11","unstructured":"Puterman, M.L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons."},{"key":"ref_12","unstructured":"Russell, S., and Norvig, P. (2021, March 10). Artificial Intelligence: A Modern Approach. Available online: https:\/\/storage.googleapis.com\/pub-tools-public-publication-data\/pdf\/27702.pdf."},{"key":"ref_13","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_14","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence 2016, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1109\/MWC.001.1900292","article-title":"Artificial intelligence enabled wireless networking for 5G and beyond: Recent advances and future challenges","volume":"27","author":"Wang","year":"2020","journal-title":"IEEE Wirel. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kulin, M., Kazaz, T., De Poorter, E., and Moerman, I. (2021). A survey on machine learning-based performance improvement of wireless networks: PHY, MAC and network layer. Electronics, 10.","DOI":"10.3390\/electronics10030318"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Asadi, A., M\u00fcller, S., Sim, G.H., Klein, A., and Hollick, M. (2018, January 16\u201319). FML: Fast machine learning for 5G mmWave vehicular communications. Proceedings of the IEEE INFOCOM 2018\u2014IEEE Conference on Computer Communications, Honolulu, HI, USA.","DOI":"10.1109\/INFOCOM.2018.8485876"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4308","DOI":"10.1109\/JIOT.2018.2853661","article-title":"A Machine Learning-Based Algorithm for Joint Scheduling and Power Control in Wireless Networks","volume":"5","author":"Cao","year":"2018","journal-title":"IEEE Internet Things J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2224","DOI":"10.1109\/COMST.2019.2904897","article-title":"Deep learning in mobile and wireless networking: A survey","volume":"21","author":"Zhang","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Khan, M.A., and Kim, J. (2020). Toward Developing Efficient Conv-AE-Based Intrusion Detection System Using Heterogeneous Dataset. Electronics, 9.","DOI":"10.3390\/electronics9111771"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3133","DOI":"10.1109\/COMST.2019.2916583","article-title":"Applications of deep reinforcement learning in communications and networking: A survey","volume":"21","author":"Luong","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/MVT.2019.2903655","article-title":"Deep reinforcement learning for mobile 5G and beyond: Fundamentals, applications, and challenges","volume":"14","author":"Xiong","year":"2019","journal-title":"IEEE Veh. Technol. Mag."},{"key":"ref_24","unstructured":"Wang, J., Zhao, L., Liu, J., and Kato, N. (2019). Smart resource allocation for mobile edge computing: A deep reinforcement learning approach. IEEE Trans. Emerg. Top. Comput."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Orsino, A., Galinina, O., Andreev, S., Yilmaz, O.N., Tirronen, T., Torsner, J., and Koucheryavy, Y. (2018, January 20\u201324). Improving initial access reliability of 5G mmWave cellular in massive V2X communications scenarios. Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, MO, USA.","DOI":"10.1109\/ICC.2018.8422158"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Galinina, O., Turlikov, A., Andreev, S., and Koucheryavy, Y. (2017, January 25\u201330). Multi-channel random access with replications. Proceedings of the 2017 IEEE International Symposium on Information Theory (ISIT), Aachen, Germany.","DOI":"10.1109\/ISIT.2017.8006987"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e3","DOI":"10.1002\/itl2.3","article-title":"A look at random access for machine-type communications in 5th generation cellular networks","volume":"1","author":"Grassi","year":"2018","journal-title":"Internet Technol. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"101883","DOI":"10.1016\/j.adhoc.2019.101883","article-title":"Pool resource management based on early collision detection in random access of massive MTC over LTE","volume":"91","author":"Alavikia","year":"2019","journal-title":"Ad Hoc Netw."},{"key":"ref_29","unstructured":"Sinitsyn, I.E., Zaripova, E.R., Gaidamaka, Y.V., and Shorgin, V.S. (2018). Success Access Probability Analysis Using Virtual Preambles Via Random Access Channel. CEUR Workshop Proceedings, CEUR."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"9347","DOI":"10.1109\/TVT.2018.2847301","article-title":"Design and Analysis of Random Access for Standalone LTE-U Systems","volume":"67","author":"Yuan","year":"2018","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Agiwal, M., Qu, M., and Jin, H. (2018, January 12\u201314). Abstraction of Random Access Procedure for Bursty MTC Traffic in 5G Networks. Proceedings of the 2018 24th Asia-Pacific Conference on Communications (APCC), Ningbo, China.","DOI":"10.1109\/APCC.2018.8633570"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Park, S., Lee, S., and Choi, W. (2019, January 18\u201321). Markov Chain Analysis for Compressed Sensing based Random Access in Cellular Systems. Proceedings of the 2019 International Conference on Computing, Networking and Communications (ICNC), Honolulu, HI, USA.","DOI":"10.1109\/ICCNC.2019.8685521"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Bekele, Y.Z., and Choi, Y.J. (2018, January 17\u201319). Scheduling for Machine Type Communications in LTE Systems. Proceedings of the 2018 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea.","DOI":"10.1109\/ICTC.2018.8539375"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Vilgelm, M., Schiessl, S., Al-Zubaidy, H., Kellerer, W., and Gross, J. (2018, January 20\u201324). On the reliability of LTE random access: Performance bounds for machine-to-machine burst resolution time. Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, MO, USA.","DOI":"10.1109\/ICC.2018.8422323"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Lee, J.Y., Noh, H., Lee, K., and Choi, J. (2018, January 3\u20136). Comparison of one-shot and handshaking systems for MTC in 5G. Proceedings of the 2018 IEEE 87th Vehicular Technology Conference (VTC Spring), Porto, Portugal.","DOI":"10.1109\/VTCSpring.2018.8417629"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2374","DOI":"10.1109\/TWC.2019.2903479","article-title":"Two-Phase Random Access Procedure for LTE-A Networks","volume":"18","author":"Cheng","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_37","unstructured":"3GPP (2014). Study on RAN Improvements for Machine-Type Communications, SGPP. Technical Report, TR 37.868."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1164","DOI":"10.1109\/JSAC.2014.2328154","article-title":"Millimeter wave channel modeling and cellular capacity evaluation","volume":"32","author":"Akdeniz","year":"2014","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Bekele, Y.Z., and June-Choi, Y. (2020, January 7\u201310). Access Point Selection Using Reinforcement Learning in Dense Mobile Networks. Proceedings of the 2020 International Conference on Information Networking (ICOIN), Barcelona, Spain.","DOI":"10.1109\/ICOIN48656.2020.9016560"},{"key":"ref_40","unstructured":"NSNAM (2021, March 10). Ns-3: A Discrete-Event Network Simulator for Internet Systems, 2006\u20132020. Available online: https:\/\/www.nsnam.org\/."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3210\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:57:22Z","timestamp":1760162242000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3210"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,5]]},"references-count":40,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21093210"],"URL":"https:\/\/doi.org\/10.3390\/s21093210","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,5]]}}}