{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T08:32:25Z","timestamp":1770971545379,"version":"3.50.1"},"reference-count":32,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,4,30]],"date-time":"2024-04-30T00:00:00Z","timestamp":1714435200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003593","name":"National Council for Scientific and Technological Development (CNPq) of Brazil","doi-asserted-by":"publisher","award":["310681\/2019-7"],"award-info":[{"award-number":["310681\/2019-7"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003593","name":"National Council for Scientific and Technological Development (CNPq) of Brazil","doi-asserted-by":"publisher","award":["314618\/2023-6"],"award-info":[{"award-number":["314618\/2023-6"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003593","name":"National Council for Scientific and Technological Development (CNPq) of Brazil","doi-asserted-by":"publisher","award":["88881.883505\/2023-01"],"award-info":[{"award-number":["88881.883505\/2023-01"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior\u2014Brasil","award":["310681\/2019-7"],"award-info":[{"award-number":["310681\/2019-7"]}]},{"name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior\u2014Brasil","award":["314618\/2023-6"],"award-info":[{"award-number":["314618\/2023-6"]}]},{"name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior\u2014Brasil","award":["88881.883505\/2023-01"],"award-info":[{"award-number":["88881.883505\/2023-01"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JSAN"],"abstract":"<jats:p>The expected huge number of connected devices in Internet of Things (IoT) applications characterizes the massive machine-type communication (mMTC) scenario, one prominent use case of beyond fifth-generation (B5G) systems. To meet mMTC connectivity requirements, grant-free (GF) random access (RA) protocols are seen as a promising solution due to the small amount of data that MTC devices usually transmit. In this paper, we propose a GF RA protocol based on a multi-agent reinforcement learning approach, applied to aid IoT devices in selecting the least congested RA pilots. The rewards obtained by the devices in collision cases resemble the congestion level of the chosen pilot. To enable the operation of the proposed method in a realistic B5G network scenario and aiming to reduce signaling overheads and centralized processing, the rewards in our proposed method are computed by the devices taking advantage of a large number of base station antennas. Numerical results demonstrate the superior performance of the proposed method in terms of latency, network throughput, and per-device throughput compared with other protocols.<\/jats:p>","DOI":"10.3390\/jsan13030030","type":"journal-article","created":{"date-parts":[[2024,4,30]],"date-time":"2024-04-30T04:01:52Z","timestamp":1714449712000},"page":"30","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A Multi-Agent Reinforcement Learning-Based Grant-Free Random Access Protocol for mMTC Massive MIMO Networks"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1850-9898","authenticated-orcid":false,"given":"Felipe Augusto Dutra","family":"Bueno","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering, Faculty of Engineering, McMaster University, 1280 Main St W, Hamilton, ON L8S 4L8, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7978-6664","authenticated-orcid":false,"given":"Alessandro","family":"Goedtel","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Federal University of Technology-Parana, Av. Alberto Carazzai, 1640, Cornelio Procopio 86300 000, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taufik","family":"Abr\u00e3o","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, State University of Londrina, Rod. Celso Garcia Cid-PR445, Londrina 86057 970, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0980-8220","authenticated-orcid":false,"given":"Jos\u00e9 Carlos","family":"Marinello","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, Federal University of Technology-Parana, Av. Alberto Carazzai, 1640, Cornelio Procopio 86300 000, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"615","DOI":"10.1109\/JSAC.2020.3019724","article-title":"Massive Access for 5G and Beyond","volume":"39","author":"Chen","year":"2021","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1109\/MWC.121.2100135","article-title":"Grant-Free Random Access in Machine-Type Communication: Approaches and Challenges","volume":"29","author":"Choi","year":"2022","journal-title":"IEEE Wirel. Commun."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"67646","DOI":"10.1109\/ACCESS.2020.2985932","article-title":"IoT Connectivity Technologies and Applications: A Survey","volume":"8","author":"Ding","year":"2020","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.dsp.2019.06.007","article-title":"Massive MIMO is a Reality-What is Next?: Five Promising Research Directions for Antenna Arrays","volume":"94","author":"Sanguinetti","year":"2019","journal-title":"Digit. Signal Process."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3590","DOI":"10.1109\/TWC.2010.092810.091092","article-title":"Noncooperative Cellular Wireless with Unlimited Numbers of Base Station Antennas","volume":"9","author":"Marzetta","year":"2010","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1109\/MCOM.2016.7402270","article-title":"Massive MIMO: Ten Myths and One Critical Question","volume":"54","author":"Larsson","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2189","DOI":"10.1109\/TMC.2019.2919839","article-title":"Recycling Random Access Opportunities with Secondary Access Class Barring","volume":"19","author":"Lee","year":"2020","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2220","DOI":"10.1109\/TWC.2017.2660489","article-title":"A Random Access Protocol for Pilot Allocation in Crowded Massive MIMO Systems","volume":"16","author":"Larsson","year":"2017","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"11111","DOI":"10.1109\/TVT.2021.3111627","article-title":"A Novel Grant-Based Pilot Access Scheme for Crowded Massive MIMO Systems","volume":"70","author":"Han","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4094","DOI":"10.1109\/TVT.2019.2898026","article-title":"Collision Resolution Protocol via Soft Decision Retransmission Criterion","volume":"68","author":"Marinello","year":"2019","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"7348","DOI":"10.1109\/TWC.2017.2747145","article-title":"A Graph-Based Random Access Protocol for Crowded Massive MIMO Systems","volume":"16","author":"Han","year":"2017","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1109\/LWC.2019.2960247","article-title":"Achieving Fair Random Access Performance in Massive MIMO Crowded Machine-Type Networks","volume":"9","author":"Marinello","year":"2020","journal-title":"IEEE Wirel. Commun. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1408","DOI":"10.1109\/TWC.2007.348337","article-title":"Contention Resolution Diversity Slotted ALOHA (CRDSA): An Enhanced Random Access Scheme for Satellite Access Packet Networks","volume":"6","author":"Casini","year":"2007","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1109\/TCOMM.2010.120710.100054","article-title":"Graph-Based Analysis and Optimization of Contention Resolution Diversity Slotted ALOHA","volume":"59","author":"Liva","year":"2011","journal-title":"IEEE Trans. Commun."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7337","DOI":"10.1109\/TCOMM.2022.3213279","article-title":"Massive Grant-Free Access with Massive MIMO and Spatially Coupled Replicas","volume":"70","author":"Valentini","year":"2022","journal-title":"IEEE Trans. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4665","DOI":"10.1109\/TCOMM.2023.3277891","article-title":"Interference Cancellation Algorithms for Grant-Free Multiple Access with Massive MIMO","volume":"71","author":"Valentini","year":"2023","journal-title":"IEEE Trans. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1109\/COMST.2019.2916177","article-title":"Toward Massive Machine Type Communications in Ultra-Dense Cellular IoT Networks: Current Issues and Machine Learning-Assisted Solutions","volume":"22","author":"Sharma","year":"2020","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"107491","DOI":"10.1016\/j.comnet.2020.107491","article-title":"Adaptive algorithm for spreading factor selection in LoRaWAN networks with multiple gateways","volume":"182","author":"Loubany","year":"2020","journal-title":"Comput. Netw."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Askhedkar, A.R., and Chaudhari, B.S. (2023). Multi-Armed Bandit Algorithm Policy for LoRa Network Performance Enhancement. J. Sens. Actuator Netw., 12.","DOI":"10.3390\/jsan12030038"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1186\/s13638-020-01783-5","article-title":"Network resource optimization with reinforcement learning for low power wide area networks","volume":"2020","author":"Park","year":"2020","journal-title":"EURASIP J. Wirel. Commun. Netw."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"30694","DOI":"10.1109\/ACCESS.2022.3160156","article-title":"D2D Assisted Q-Learning Random Access for NOMA-Based MTC Networks","volume":"10","author":"Souza","year":"2022","journal-title":"IEEE Access"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"6861","DOI":"10.1109\/TCOMM.2022.3198125","article-title":"Reinforcement Learning for NOMA-ALOHA Under Fading","volume":"70","author":"Ko","year":"2022","journal-title":"IEEE Trans. Commun."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"140270","DOI":"10.1109\/ACCESS.2023.3341585","article-title":"DRL-Based Resource Allocation for NOMA-Enabled D2D Communications Underlay Cellular Networks","volume":"11","author":"Jeong","year":"2023","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Yang, C., Wang, Y., Lan, S., and Zhu, L. (2024). Multi-agent Reinforcement Learning based Distributed Channel Access for Industrial Edge-Cloud Web 3.0. IEEE Trans. Netw. Sci. Eng., early access.","DOI":"10.1109\/TNSE.2024.3377441"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3070","DOI":"10.1109\/ACCESS.2023.3349350","article-title":"Decentralized Multi-Agent DQN-Based Resource Allocation for Heterogeneous Traffic in V2X Communications","volume":"12","author":"Lee","year":"2024","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1109\/COMST.2022.3151028","article-title":"Cellular, Wide-Area, and Non-Terrestrial IoT: A Survey on 5G Advances and the Road Toward 6G","volume":"24","author":"Vaezi","year":"2022","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_27","unstructured":"Ericsson (2023). Ericsson Mobility Report, Ericsson. Technical Report."},{"key":"ref_28","unstructured":"Bello, L.M., Mitchell, P., and Grace, D. (2014, January 14\u201316). Application of Q-Learning for RACH Access to Support M2M Traffic over a Cellular Network. Proceedings of the European Wireless 2014; 20th European Wireless Conference, Barcelona, Spain."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"108787","DOI":"10.1016\/j.comnet.2022.108787","article-title":"Throughput and latency in the distributed Q-learning random access mMTC networks","volume":"206","author":"Silva","year":"2022","journal-title":"Comput. Netw."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/LCOMM.2019.2896929","article-title":"Collaborative Distributed Q-Learning for RACH Congestion Minimization in Cellular IoT Networks","volume":"23","author":"Sharma","year":"2019","journal-title":"IEEE Commun. Lett."},{"key":"ref_31","unstructured":"(2024, March 22). 3GPP. Spatial channel model for Multiple Input Multiple Output (MIMO) simulations. Technical Report (TR) 25.996, 3rd Generation Partnership Project (3GPP), 2018. Version 15.0.0. Available online: https:\/\/portal.etsi.org\/webapp\/workprogram\/Report_WorkItem.asp?WKI_ID=55817."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"6597","DOI":"10.1109\/TWC.2022.3151329","article-title":"Exploring the Non-Overlapping Visibility Regions in XL-MIMO Random Access and Scheduling","volume":"21","author":"Marinello","year":"2022","journal-title":"IEEE Trans. Wirel. Commun."}],"container-title":["Journal of Sensor and Actuator Networks"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2224-2708\/13\/3\/30\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:36:45Z","timestamp":1760107005000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2224-2708\/13\/3\/30"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,30]]},"references-count":32,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["jsan13030030"],"URL":"https:\/\/doi.org\/10.3390\/jsan13030030","relation":{},"ISSN":["2224-2708"],"issn-type":[{"value":"2224-2708","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,30]]}}}