{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T22:28:56Z","timestamp":1783549736830,"version":"3.55.0"},"reference-count":36,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T00:00:00Z","timestamp":1670025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Chung-Ang University","award":["IITP-2022-RS-2022-00156353"],"award-info":[{"award-number":["IITP-2022-RS-2022-00156353"]}]},{"name":"Ministry of Science and ICT","award":["IITP-2022-RS-2022-00156353"],"award-info":[{"award-number":["IITP-2022-RS-2022-00156353"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this paper, a resource allocation (RA) scheme based on deep reinforcement learning (DRL) is designed for device-to-device (D2D) communications underlay cellular networks. The goal of RA is to determine the transmission power and spectrum channel of D2D links to maximize the sum of the average effective throughput of all cellular and D2D links in a cell accumulated over multiple time steps, where a cellular channel can be allocated to multiple D2D links. Allowing a cellular channel to be shared by multiple D2D links and considering performance over multiple time steps require a high level of system overhead and computational complexity so that optimal RA is practically infeasible in this scenario, especially when a large number of D2D links are involved. To mitigate the complexity, we propose a sub-optimal RA scheme based on a multi-agent DRL, which operates with shared information in participating devices, such as locations and allocated resources. Each agent corresponds to each D2D link and multiple agents perform learning in a staggered and cyclic manner. The proposed DRL-based RA scheme allocates resources to D2D devices promptly according to dynamically varying network set-ups, including device locations. The proposed sub-optimal RA scheme outperforms other schemes, where the performance gain becomes significant when the densities of devices in a cell are high.<\/jats:p>","DOI":"10.3390\/s22239459","type":"journal-article","created":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T08:10:57Z","timestamp":1670227857000},"page":"9459","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Deep Reinforcement Learning Based Resource Allocation for D2D Communications Underlay Cellular Networks"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1225-5767","authenticated-orcid":false,"given":"Seoyoung","family":"Yu","sequence":"first","affiliation":[{"name":"School of Electrical and Electronics Engineering, Chung-Ang University, Seoul 06974, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6117-7489","authenticated-orcid":false,"given":"Jeong Woo","family":"Lee","sequence":"additional","affiliation":[{"name":"School of Electrical and Electronics Engineering, Chung-Ang University, Seoul 06974, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/MCOM.2009.5350367","article-title":"Device-to-device communication as an underlay to LTE-advanced networks","volume":"47","author":"Doppler","year":"2009","journal-title":"IEEE Commun. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1109\/LCOMM.2017.2780819","article-title":"Resource Allocation for Multiple Device-to-Device Cluster Multicast Communications Underlay Cellular Networks","volume":"22","author":"Kim","year":"2018","journal-title":"IEEE Commun. Lett."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"8357","DOI":"10.1109\/TVT.2017.2691470","article-title":"Optimal Resource Allocation in Multicast Device-to-Device Communications Underlaying LTE Networks","volume":"66","author":"Meshgi","year":"2017","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/MCOM.2014.6807946","article-title":"Device-to-device communications in cellular networks","volume":"52","author":"Feng","year":"2014","journal-title":"IEEE Commun. Mag."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"20632","DOI":"10.1109\/ACCESS.2019.2895975","article-title":"Joint Resource Allocation and Power Control Algorithm for Cooperative D2D Heterogeneous Networks","volume":"7","author":"Gao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1159","DOI":"10.1109\/TVT.2016.2553124","article-title":"Resource Allocation for D2D Communications Underlay in Rayleigh Fading Channels","volume":"66","author":"Wang","year":"2017","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_7","first-page":"2097","article-title":"Energy-Efficient Resource Reuse Scheme for D2D Communications Underlaying Cellular Networks","volume":"21","author":"Hu","year":"2017","journal-title":"IEEE Commun. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1109\/LCOMM.2017.2669179","article-title":"A Signaling-Based Incentive Mechanism for Device-to-Device Content Sharing in Cellular Networks","volume":"21","author":"Zhang","year":"2017","journal-title":"IEEE Commun. Lett."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1109\/TWC.2011.122010.100963","article-title":"Reliability Improvement Using Receive Mode Selection in the Device-to-Device Uplink Period Underlaying Cellular Networks","volume":"10","author":"Min","year":"2011","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Nguyen, T., Nguyen, V., Nguyen, H., Tu, L., Van Chien, T., and Nguyen, T. (2022). On the Performance of Underlay Device-to-Device Communications. Sensors, 22.","DOI":"10.3390\/s22041456"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1109\/LCOMM.2019.2907252","article-title":"Performance Analysis and Resource Allocation for Cooperative D2D Communication in Cellular Networks With Multiple D2D Pairs","volume":"23","author":"Lee","year":"2019","journal-title":"IEEE Commun. Lett."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2182","DOI":"10.1109\/TVT.2015.2424395","article-title":"Joint Spectrum and Power Allocation for D2D Communications Underlaying Cellular Networks","volume":"65","author":"Yin","year":"2016","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1109\/TSIPN.2017.2742941","article-title":"Power Allocation for Underlay Device-to-Device Communication Over Multiple Channels","volume":"4","author":"AliHemmati","year":"2018","journal-title":"IEEE Trans. Signal Inf. Process. Over Netw."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1008","DOI":"10.1109\/LWC.2019.2903798","article-title":"Resource Allocation for D2D Communication With Multiple D2D Pairs Reusing Multiple Channels","volume":"8","author":"Mach","year":"2019","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"16424","DOI":"10.1109\/ACCESS.2021.3132260","article-title":"Resource Allocation for D2D Cellular Networks With QoS Constraints: A DC Programming- Based Approach","volume":"10","author":"Chang","year":"2022","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1398","DOI":"10.1109\/LCOMM.2015.2444403","article-title":"Resource Allocation for Device-to-Device Communication Underlaying Cellular Networks: An Alternating Optimization Method","volume":"19","author":"Zhao","year":"2015","journal-title":"IEEE Commun. Lett."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"887","DOI":"10.1109\/LCOMM.2020.3042490","article-title":"Resource Allocation Scheme for Guarantee of QoS in D2D Communications Using Deep Neural Network","volume":"25","author":"Lee","year":"2021","journal-title":"IEEE Commun. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zheng, Z., Chi, Y., Ding, G., and Yu, G. (2022). Deep-Learning-Based Resource Allocation for Time-Sensitive Device-to-Device Networks. Sensors, 22.","DOI":"10.3390\/s22041551"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"5235","DOI":"10.1109\/TWC.2021.3138733","article-title":"Deep learning-based resource allocation for device-to-device communication","volume":"21","author":"Lee","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Xu, Y.-H., Zhou, W., Zhang, Y.-G., and Yu, G. (2022). Stochastic game for Resource Management in cellular zero-touch deterministic industrial M2M networks. IEEE Wirel. Commun. Lett., 1.","DOI":"10.1109\/LSENS.2022.3194141"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Xu, Y.-H., Li, J.-H., Zhou, W., and Chen, C. (2022). Learning-Empowered Resource Allocation for Air Slicing in UAV-Assisted Cellular V2X Communications. IEEE Syst. J., 1\u20134.","DOI":"10.1109\/JSYST.2022.3144159"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Park, H., and Lim, Y. (2020). Reinforcement Learning for Energy Optimization with 5G Communications in Vehicular Social Networks. Sensors, 20.","DOI":"10.3390\/s20082361"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"6452","DOI":"10.1109\/TVT.2020.2985873","article-title":"Energy-efficient power allocation and Q-learning-based relay selection for relay-aided D2D communication","volume":"69","author":"Wang","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1610","DOI":"10.1109\/TWC.2019.2894403","article-title":"Multi-agent reinforcement learning for efficient content caching in Mobile D2D Networks","volume":"18","author":"Jiang","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/LCOMM.2021.3079920","article-title":"Deep Reinforcement Learning-Based Dynamic Spectrum Access for D2D Communication Underlay Cellular Networks","volume":"25","author":"Huang","year":"2021","journal-title":"IEEE Commun. Lett."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1109\/LWC.2019.2958814","article-title":"A Deep Reinforcement Learning Based D2D Relay Selection and Power Level Allocation in mmWave Vehicular Networks","volume":"9","author":"Zhang","year":"2020","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"11121","DOI":"10.1109\/TVT.2021.3106398","article-title":"DRL-Based Sum-Rate Maximization in D2D Communication Underlaid Uplink Cellular Networks","volume":"70","author":"Ron","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3163","DOI":"10.1109\/TVT.2019.2897134","article-title":"Deep Reinforcement Learning Based Resource Allocation for V2V Communications","volume":"68","author":"Ye","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"164533","DOI":"10.1109\/ACCESS.2019.2952411","article-title":"Distributed Deep Deterministic Policy Gradient for Power Allocation Control in D2D-Based V2V Communications","volume":"7","author":"Nguyen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2943","DOI":"10.1109\/LCOMM.2021.3093385","article-title":"Adaptive Reinforcement Learning Framework for NOMA-UAV Networks","volume":"25","author":"Mahmud","year":"2021","journal-title":"IEEE Commun. Lett."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1828","DOI":"10.1109\/TVT.2019.2961405","article-title":"Multi-Agent Deep Reinforcement Learning Based Spectrum Allocation for D2D Underlay Communications","volume":"69","author":"Li","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1659","DOI":"10.1109\/LWC.2022.3170998","article-title":"Multi-Agent Deep Reinforcement Learning-Based Power Control and Resource Allocation for D2D Communications","volume":"11","author":"Xiang","year":"2022","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1016\/j.dcan.2021.09.013","article-title":"Deep reinforcement learning-based resource allocation for D2D Communications in Heterogeneous Cellular Networks","volume":"8","author":"Zhi","year":"2021","journal-title":"Digit. Commun. Netw."},{"key":"ref_35","unstructured":"Zhou, X., Zhang, W., Chen, Z., Diao, S., and Zhang, T. (2021, January 6\u201314). Efficient Neural Network Training via Forward and Backward Propagation Sparsification. Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Virtual."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"3355","DOI":"10.1109\/TWC.2017.2681659","article-title":"Fundamentals of modeling finite wireless networks using binomial point process","volume":"16","author":"Afshang","year":"2017","journal-title":"IEEE Trans. Wirel. Commun."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9459\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:33:32Z","timestamp":1760146412000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9459"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,3]]},"references-count":36,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["s22239459"],"URL":"https:\/\/doi.org\/10.3390\/s22239459","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,3]]}}}