{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T19:03:04Z","timestamp":1770750184604,"version":"3.50.0"},"reference-count":55,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004952","name":"Atlantic Canada Opportunities Agency","doi-asserted-by":"publisher","award":["208437"],"award-info":[{"award-number":["208437"]}],"id":[{"id":"10.13039\/501100004952","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Efficient and reliable data routing is critical in Advanced Metering Infrastructure (AMI) within Smart Grids, dictating the overall network performance and resilience. This paper introduces Q-RPL, a novel Q-learning-based Routing Protocol designed to enhance routing decisions in AMI deployments based on wireless mesh technologies. Q-RPL leverages the principles of Reinforcement Learning (RL) to dynamically select optimal next-hop forwarding candidates, adapting to changing network conditions. The protocol operates on top of the standard IPv6 Routing Protocol for Low-Power and Lossy Networks (RPL), integrating it with intelligent decision-making capabilities. Through extensive simulations carried out in real map scenarios, Q-RPL demonstrates a significant improvement in key performance metrics such as packet delivery ratio, end-to-end delay, and compliant factor compared to the standard RPL implementation and other benchmark algorithms found in the literature. The adaptability and robustness of Q-RPL mark a significant advancement in the evolution of routing protocols for Smart Grid AMI, promising enhanced efficiency and reliability for future intelligent energy systems. The findings of this study also underscore the potential of Reinforcement Learning to improve networking protocols.<\/jats:p>","DOI":"10.3390\/s24154818","type":"journal-article","created":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T06:11:36Z","timestamp":1721887896000},"page":"4818","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Q-RPL: Q-Learning-Based Routing Protocol for Advanced Metering Infrastructure in Smart Grids"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8949-3686","authenticated-orcid":false,"given":"Carlos Lester","family":"Duenas Santos","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering Department, University of New Brunswick, Fredericton, NB E3B 5A3, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3263-0122","authenticated-orcid":false,"given":"Ahmad Mohamad","family":"Mezher","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering Department, University of New Brunswick, Fredericton, NB E3B 5A3, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7291-3584","authenticated-orcid":false,"given":"Juan Pablo","family":"Astudillo Le\u00f3n","sequence":"additional","affiliation":[{"name":"School of Mathematical and Computational Sciences, Yachay Tech University, Urcuqu\u00ed 100115, Ecuador"},{"name":"Communication Networks and Intelligent Services Research Group (ComNet Innova YT), Yachay Tech University, Urcuqu\u00ed 100115, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3674-3995","authenticated-orcid":false,"given":"Julian","family":"Cardenas Barrera","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering Department, University of New Brunswick, Fredericton, NB E3B 5A3, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9227-621X","authenticated-orcid":false,"given":"Eduardo","family":"Castillo Guerra","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering Department, University of New Brunswick, Fredericton, NB E3B 5A3, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0943-2291","authenticated-orcid":false,"given":"Julian","family":"Meng","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering Department, University of New Brunswick, Fredericton, NB E3B 5A3, Canada"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"ref_1","unstructured":"(2021). IEEE Draft Standard for Information Technology-Telecommunications and Information Exchange between Systems-Local and Metropolitan Area Networks-Specific Requirements-Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications-Amendment 10: Mesh Networking (Standard No. IEEE P802.11s Draft D)."},{"key":"ref_2","unstructured":"(2011). IEEE Standard for Local and Metropolitan Area Networks-Part 15.4: Low-Rate Wireless Personal Area Networks (LR-WPANs) Amendment 3: Physical Layer (PHY) Specifications for Low-Data-Rate, Wireless, Smart Metering Utility Networks (Standard No. IEEE 802.15.4g)."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1032","DOI":"10.1587\/transcom.2016SCI0002","article-title":"IEEE 802.15. 4g based Wi-SUN communication systems","volume":"100","author":"Harada","year":"2017","journal-title":"IEICE Trans. Commun."},{"key":"ref_4","unstructured":"Chang, K.H., and Mason, B. (2012, January 5\u20138). The IEEE 802.15. 4g standard for smart metering utility networks. Proceedings of the 2012 IEEE Third International Conference on Smart Grid Communications (SmartGridComm), Tainan, Taiwan."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Karp, B., and Kung, H.T. (2000, January 6\u201311). GPSR: Greedy Perimeter Stateless Routing for Wireless Networks. Proceedings of the 6th Annual International Conference on Mobile Computing and Networking, New York, NY, USA. MobiCom \u201900.","DOI":"10.1145\/345910.345953"},{"key":"ref_6","first-page":"1","article-title":"The collection tree protocol (CTP)","volume":"123","author":"Fonseca","year":"2006","journal-title":"TinyOS TEP"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Clausen, T., Dearlove, C., Jacquet, P., and Herberg, U. (2014). The Optimized Link State Routing Protocol Version 2, Internet Engineering Task Force (IETF). RFC 7181.","DOI":"10.17487\/rfc7181"},{"key":"ref_8","unstructured":"Winter, T., Thubert, P., Brandt, A., Hui, J., Kelsey, R., Levis, P., Pister, K., Struik, R., and Vasseur, J.P. (2012). IPv6 Routing Protocol for Low-Power and Lossy Networks, Internet Engineering Task Force (IETF). RFC 6550."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Dawson-Haggerty, S., Tavakoli, A., and Culler, D. (2010, January 4\u20136). Hydro: A hybrid routing protocol for low-power and lossy networks. Proceedings of the 2010 First IEEE International Conference on Smart Grid Communications, Gaithersburg, MD, USA.","DOI":"10.1109\/SMARTGRID.2010.5622053"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.comnet.2017.06.025","article-title":"Lightweight on-demand ad hoc distance-vector routing-next generation (LOADng): Protocol, extension, and applicability","volume":"126","author":"Clausen","year":"2017","journal-title":"Comput. Netw."},{"key":"ref_11","first-page":"802","article-title":"HWMP specification","volume":"11","author":"Joshi","year":"2006","journal-title":"IEEE P802"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"103476","DOI":"10.1016\/j.jnca.2022.103476","article-title":"RPL routing protocol over IoT: A comprehensive survey, recent advances, insights, bibliometric analysis, recommendations, and future directions","volume":"207","author":"Darabkh","year":"2022","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Iyer, G., Agrawal, P., Monnerie, E., and Cardozo, R.S. (2011, January 17\u201320). Performance analysis of wireless mesh routing protocols for smart utility networks. Proceedings of the 2011 IEEE International Conference on Smart Grid Communications (SmartGridComm), Brussels, Belgium.","DOI":"10.1109\/SmartGridComm.2011.6102301"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Iyer, G., Agrawal, P., and Cardozo, R.S. (2013, January 9\u201313). Performance comparison of routing protocols over smart utility networks: A simulation study. Proceedings of the 2013 IEEE Globecom Workshops (GC Wkshps), Atlanta, GA, USA.","DOI":"10.1109\/GLOCOMW.2013.6825116"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ho, Q.D., Gao, Y., Rajalingham, G., and Le-Ngoc, T. (2014, January 10\u201314). Performance and applicability of candidate routing protocols for smart grid\u2019s wireless mesh neighbor area networks. Proceedings of the 2014 IEEE International Conference on Communications (ICC), Sydney, NSW, Australia.","DOI":"10.1109\/ICC.2014.6883894"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1607","DOI":"10.1109\/COMST.2018.2874356","article-title":"A Survey of Limitations and Enhancements of the IPv6 Routing Protocol for Low-Power and Lossy Networks: A Focus on Core Operations","volume":"21","author":"Ghaleb","year":"2019","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"102001","DOI":"10.1016\/j.adhoc.2019.102001","article-title":"A comprehensive survey on enhancements and limitations of the RPL protocol: A focus on the objective function","volume":"96","author":"Lamaazi","year":"2020","journal-title":"Hoc Netw."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"57401","DOI":"10.1109\/ACCESS.2023.3283208","article-title":"ML-RPL: Machine Learning-based routing protocol for Wireless Smart Grid Networks","volume":"11","author":"Santos","year":"2023","journal-title":"IEEE Access"},{"key":"ref_19","unstructured":"Mezher, A.M., Due\u00f1as Santos, C.L., Rebollo-Monedero, D., C\u00e1rdenas-Barrera, J., Aguilar Igartua, M., Meng, J., and Castillo Guerra, E. (November, January 30). GNB-RPL: Gaussian Na\u00efve Bayes for RPL Routing Protocol in Smart Grid Communications. Proceedings of the 19th ACM International Symposium on QoS and Security for Wireless and Mobile Networks, Montreal, QC, Canada."},{"key":"ref_20","unstructured":"Mezher, A.M., Due\u00f1as Santos, C.L., Astudillo Leon, J.P., C\u00e1rdenas-Barrera, J., Meng, J., and Castillo Guerra, E. (November, January 30). Are ML Models Scenario-Independent in Enhancing Routing Efficiency for Smart Grid Networks?. Proceedings of the Int\u2019l ACM Symposium on Performance Evaluation of Wireless Ad Hoc, Sensor, & Ubiquitous Networks, New York, NY, USA. PE-WASUN \u201923."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pister, K., Dejean, N., and Barthel, D. (2012). Routing Metrics Used for Path Calculation in Low-Power and Lossy Networks, Internet Engineering Task Force (IETF). RFC 6551; RFC 7181.","DOI":"10.17487\/rfc6551"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Thubert, E.P. (2012). Objective Function Zero for the Routing Protocol for Low-Power and Lossy Networks (RPL), Internet Engineering Task Force (IETF). RFC 6552.","DOI":"10.17487\/rfc6552"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gnawali, P.L.O. (2012). The Minimum Rank with Hysteresis Objective Function, Internet Engineering Task Force (IETF). RFC 6719.","DOI":"10.17487\/rfc6719"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1007\/s11036-020-01662-9","article-title":"Using Multiple RPL Instances to Enhance the Performance of New 6G and Internet of Everything (6G\/IoE)-Based Healthcare Monitoring Systems","volume":"26","author":"Mardini","year":"2021","journal-title":"Mob. Netw. Appl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"96686","DOI":"10.1109\/ACCESS.2020.2995794","article-title":"Multi-Topology Based QoS-Differentiation in RPL for Internet of Things Applications","volume":"8","author":"Bhandari","year":"2020","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13638-020-1645-4","article-title":"Routing protocol for Low-Power and Lossy Networks for heterogeneous traffic network","volume":"2020","author":"Musaddiq","year":"2020","journal-title":"Eurasip J. Wirel. Commun. Netw."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"60163","DOI":"10.1109\/ACCESS.2021.3074436","article-title":"WRF-RPL: Weighted Random Forward RPL for High Traffic and Energy Demanding Scenarios","volume":"9","author":"Acevedo","year":"2021","journal-title":"IEEE Access"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mishra, S.N., and Khatua, M. (November, January 26). Achieving Hard Reliability in RPL for Mission-Critical IoT Applications. Proceedings of the 2022 IEEE 8th World Forum on Internet of Things (WF-IoT), Yokohama, Japan.","DOI":"10.1109\/WF-IoT54382.2022.10152158"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Gaddour, O., Koub\u01cea, A., Baccour, N., and Abid, M. (2014, January 12\u201316). OF-FL: QoS-aware fuzzy logic objective function for the RPL routing protocol. Proceedings of the 2014 12th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks, WiOpt 2014, Hammamet, Tunisia.","DOI":"10.1109\/WIOPT.2014.6850321"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Harshavardhana, T.G., Vineeth, B.S., Anand, S.V., and Hegde, M. (2018, January 3\u20137). Power control and cross-layer design of RPL objective function for low power and lossy networks. Proceedings of the 2018 10th International Conference on Communication Systems and Networks, COMSNETS 2018, Bengaluru, India.","DOI":"10.1109\/COMSNETS.2018.8328200"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"117593","DOI":"10.1016\/j.eswa.2022.117593","article-title":"An innovative RPL objective function for broad range of IoT domains utilizing fuzzy logic and multiple metrics","volume":"205","author":"Darabkh","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1007\/s42979-024-02694-8","article-title":"Data Dissemination Framework for Optimizing Overhead in IoT-Enabled Systems Using Tabu-RPL","volume":"5","author":"Prajapati","year":"2024","journal-title":"SN Comput. Sci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2621","DOI":"10.1007\/s00500-023-09564-0","article-title":"Trickle timer modification for RPL in Internet of things","volume":"28","author":"Shetty","year":"2024","journal-title":"Soft Comput."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Duenas Santos, C.L., Astudillo Le\u00f3n, J.P., Mezher, A.M., Cardenas Barrera, J., Meng, J., and Castillo Guerra, E. (2022, January 24\u201328). RPL+: An Improved Parent Selection Strategy for RPL in Wireless Smart Grid Networks. Proceedings of the 19th ACM International Symposium on Performance Evaluation of Wireless Ad Hoc, Sensor, & Ubiquitous Networks, Montreal, QC, Canada.","DOI":"10.1145\/3551663.3558677"},{"key":"ref_35","unstructured":"Raschka, S., and Mirjalili, V. (2017). Python Machine Learning: Machine Learning and Deep Learning with Python, Scikit-Learn, and TensorFlow, Packt Publishing Ltd."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"3072","DOI":"10.1109\/COMST.2019.2924243","article-title":"Application of machine learning in wireless networks: Key techniques and open issues","volume":"21","author":"Sun","year":"2019","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"52523","DOI":"10.1109\/ACCESS.2021.3069210","article-title":"Applications of machine learning in networking: A survey of current issues and future challenges","volume":"9","author":"Ridwan","year":"2021","journal-title":"IEEE Access"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1578","DOI":"10.1109\/COMST.2021.3073009","article-title":"Survey on machine learning for intelligent end-to-end communication toward 6G: From network access, routing to traffic control and streaming adaption","volume":"23","author":"Tang","year":"2021","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kim, B.S., Suh, B., Seo, I.J., Lee, H.B., Gong, J.S., and Kim, K.I. (2023). An Enhanced Tree Routing Based on Reinforcement Learning in Wireless Sensor Networks. Sensors, 23.","DOI":"10.3390\/s23010223"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"7691","DOI":"10.1007\/s11227-023-05724-z","article-title":"RI-RPL: A new high-quality RPL-based routing protocol using Q-learning algorithm","volume":"80","author":"Zahedy","year":"2023","journal-title":"J. Supercomput."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1007\/s11235-023-01075-5","article-title":"QFS-RPL: Mobility and energy aware multi path routing protocol for the internet of mobile things data transfer infrastructures","volume":"85","author":"Alilou","year":"2024","journal-title":"Telecommun. Syst."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Rabet, I., Fotouhi, H., Alves, M., Vahabi, M., and Bj\u00f6rkman, M. (2024). ACTOR: Adaptive Control of Transmission Power in RPL. Sensors, 24.","DOI":"10.3390\/s24072330"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_44","unstructured":"Raschka, S., Liu, Y.H., Mirjalili, V., and Dzhulgakov, D. (2022). Machine Learning with PyTorch and Scikit-Learn: Develop Machine Learning and Deep Learning Models with Python, Packt Publishing Ltd."},{"key":"ref_45","unstructured":"(2020). IEEE Standard for Low-Rate Wireless Networks (Standard No. IEEE Std 802.15.4-2020 (Revision of IEEE Std 802.15.4-2015))."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1016\/j.comnet.2017.07.001","article-title":"DT-RPL: Diverse bidirectional traffic delivery through RPL routing protocol in low power and lossy networks","volume":"126","author":"Kim","year":"2017","journal-title":"Comput. Netw."},{"key":"ref_47","unstructured":"(2024, July 10). OMNeT++ Discrete Event Simulator. Available online: https:\/\/omnetpp.org\/."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"22938","DOI":"10.1109\/ACCESS.2024.3362889","article-title":"Investigating and Analyzing Simulation Tools of Wireless Sensor Networks: A Comprehensive Survey","volume":"12","author":"Adday","year":"2024","journal-title":"IEEE Access"},{"key":"ref_49","unstructured":"(2024, April 29). The ns-3 Network Simulator Project. ns-3 Network Simulator. Available online: https:\/\/www.nsnam.org\/."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Bartolozzi, L., Pecorella, T., and Fantacci, R. (2012, January 19\u201323). ns-3 RPL module: IPv6 routing protocol for low power and lossy networks. Proceedings of the 5th International ICST Conference on Simulation Tools and Techniques, Desenzano del Garda, Italy.","DOI":"10.4108\/icst.simutools.2012.247718"},{"key":"ref_51","unstructured":"Chen, Y.b., Hou, K.M., Chanet, J.P., and El Gholami, K. (2013, January 18\u201320). A RPL based Adaptive and Scalable Data-collection Protocol module for NS-3 simulation platform. Proceedings of the NICST 2103 New Information Communication Science and Technology for Sustainable Development: France-China International Workshop, Clermont-Ferrand, France."},{"key":"ref_52","unstructured":"El Ghomali, K., Elkamoun, N., Hou, K.M., Chen, Y., Chanet, J.P., and Li, J.J. (2013, January 18\u201320). A new WPAN Model for NS-3 simulator. Proceedings of the NICST\u20192103 New Information Communication Science and Technology for Sustainable Development: France-China International Workshop, Clermont-Ferrand, France."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"119648","DOI":"10.1109\/ACCESS.2021.3107144","article-title":"Sub-1 ghz frequency band wireless coexistence for the internet of things","volume":"9","author":"Nagai","year":"2021","journal-title":"IEEE Access"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"216805","DOI":"10.1109\/ACCESS.2020.3041690","article-title":"Predictive Traffic Control and Differentiation on Smart Grid Neighborhood Area Networks","volume":"8","author":"Leon","year":"2020","journal-title":"IEEE Access"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"109956","DOI":"10.1016\/j.comnet.2023.109956","article-title":"Exploring the potential, limitations, and future directions of wireless technologies in smart grid networks: A comparative analysis","volume":"235","author":"Santos","year":"2023","journal-title":"Comput. Netw."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/15\/4818\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:22:57Z","timestamp":1760109777000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/15\/4818"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":55,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["s24154818"],"URL":"https:\/\/doi.org\/10.3390\/s24154818","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,25]]}}}