{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T15:12:20Z","timestamp":1782313940179,"version":"3.54.5"},"reference-count":54,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2020,7,26]],"date-time":"2020-07-26T00:00:00Z","timestamp":1595721600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2018R1D1A1A09082266"],"award-info":[{"award-number":["2018R1D1A1A09082266"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The next generation of the Internet of Things (IoT) networks is expected to handle a massive scale of sensor deployment with radically heterogeneous traffic applications, which leads to a congested network, calling for new mechanisms to improve network efficiency. Existing protocols are based on simple heuristics mechanisms, whereas the probability of collision is still one of the significant challenges of future IoT networks. The medium access control layer of IEEE 802.15.4 uses a distributed coordination function to determine the efficiency of accessing wireless channels in IoT networks. Similarly, the network layer uses a ranking mechanism to route the packets. The objective of this study was to intelligently utilize the cooperation of multiple communication layers in an IoT network. Recently, Q-learning (QL), a machine learning algorithm, has emerged to solve learning problems in energy and computational-constrained sensor devices. Therefore, we present a QL-based intelligent collision probability inference algorithm to optimize the performance of sensor nodes by utilizing channel collision probability and network layer ranking states with the help of an accumulated reward function. The simulation results showed that the proposed scheme achieved a higher packet reception ratio, produces significantly lower control overheads, and consumed less energy compared to current state-of-the-art mechanisms.<\/jats:p>","DOI":"10.3390\/s20154158","type":"journal-article","created":{"date-parts":[[2020,7,27]],"date-time":"2020-07-27T04:39:50Z","timestamp":1595824790000},"page":"4158","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Reinforcement Learning-Enabled Cross-Layer Optimization for Low-Power and Lossy Networks under Heterogeneous Traffic Patterns"],"prefix":"10.3390","volume":"20","author":[{"given":"Arslan","family":"Musaddiq","sequence":"first","affiliation":[{"name":"Department of Information and Communication Engineering, Yeungnam University, 280 Daehak-Ro, Gyeongsan, Gyeongbuk 38541, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zulqar","family":"Nain","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Yeungnam University, 280 Daehak-Ro, Gyeongsan, Gyeongbuk 38541, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yazdan","family":"Ahmad Qadri","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Yeungnam University, 280 Daehak-Ro, Gyeongsan, Gyeongbuk 38541, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9756-1909","authenticated-orcid":false,"given":"Rashid","family":"Ali","sequence":"additional","affiliation":[{"name":"School of Intelligent Mechatronics Engineering, Sejong University, Seoul 05006, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8454-6980","authenticated-orcid":false,"given":"Sung Won","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Yeungnam University, 280 Daehak-Ro, Gyeongsan, Gyeongbuk 38541, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,7,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/MWC.2017.1600421","article-title":"Internet of Things architecture: Recent advances, taxonomy, requirements, and open challenges","volume":"24","author":"Yaqoob","year":"2017","journal-title":"IEEE Wireless Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8459","DOI":"10.1109\/ACCESS.2018.2808324","article-title":"A survey on resource management in IoT operating systems","volume":"6","author":"Musaddiq","year":"2018","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Naeem, M.A., Ali, R., Kim, B.-S., Nor, S.A., and Hassan, S. (2018). A periodic caching strategy solution for the smart city in information-centric Internet of Things. Sustainability, 10.","DOI":"10.3390\/su10072576"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1109\/COMST.2020.2973314","article-title":"The Future of Healthcare Internet of Things: A Survey of Emerging Technologies","volume":"22","author":"Qadri","year":"2020","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_5","first-page":"1","article-title":"Q-learning- enabled channel access in next-generation dense wireless networks for IoT-based eHealth systems","volume":"178","author":"Ali","year":"2019","journal-title":"J. Wirel. Commun. Netw."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2233","DOI":"10.1109\/TII.2014.2300753","article-title":"Internet of Things in industries: A survey","volume":"10","author":"Xu","year":"2014","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1109\/TII.2011.2166794","article-title":"Smart grid technologies: Communication technologies and standards","volume":"7","author":"Gungor","year":"2011","journal-title":"IEEE Trans. Ind. Inf."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3093","DOI":"10.1109\/TII.2019.2903100","article-title":"Access control and resource allocation for M2M communications in industrial automation","volume":"15","author":"Zhou","year":"2019","journal-title":"IEEE Trans. Ind. Informat."},{"key":"ref_9","unstructured":"Alexander, R., Brandt, A., Vasseur, J., Hui, J., Pister, K., Thubert, P., Levis, P., Struik, R., Kelsey, R., and Winter, T. (2012). RPL: IPv6 Routing Protocol for Low-Power and Lossy Networks, Internet Engineering Task Force (IETF). RFC 6550."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gnawali, O., and Levis, P. (2012). The Minimum Rank with Hysteresis Objective Function, Internet Engineering Task Force (IETF). RFC 6719.","DOI":"10.17487\/rfc6719"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Thubert, P. (2012). Objective Function Zero for the Routing Protocol for Low-Power and Lossy Networks (RPL), Internet Engineering Task Force (IETF). RFC 6552.","DOI":"10.17487\/rfc6552"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1109\/JIOT.2015.2394438","article-title":"A survey of MAC layer issues and protocols for machine-to-machine communications","volume":"2","author":"Rajandekar","year":"2015","journal-title":"IEEE Internet Things J."},{"key":"ref_13","first-page":"1","article-title":"Routing protocol for Low-Power and Lossy Networks for heterogeneous traffic network","volume":"2020","author":"Musaddiq","year":"2020","journal-title":"J. Wirel. Commun. Netw."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Khomami, P., Khomami, G., and Fontan, F. (2018). The relation between the probability of collision-free broadcast transmission in a wireless network and the stirling number of the second kind. Mathematics, 6.","DOI":"10.3390\/math6070127"},{"key":"ref_15","unstructured":"Alpaydm, E. (2014). Introduction to Machine Learning, MIT Press. [3rd ed.]."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.dcan.2017.10.002","article-title":"Machine learning for Internet of Things data analysis: A survey","volume":"4","author":"Mahdavinejad","year":"2018","journal-title":"Digit. Commun. Netw."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4260","DOI":"10.1109\/JIOT.2019.2963371","article-title":"Learning-based context-aware resource allocation for edge computing-empowered industrial IoT","volume":"7","author":"Liao","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1996","DOI":"10.1109\/COMST.2014.2320099","article-title":"Machine learning in wireless sensor networks: Algorithms strategies and applications","volume":"16","author":"Alsheikh","year":"2014","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press. [2nd ed.].","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1109\/TMC.2016.2585107","article-title":"Load balancing under heavy traffic in rpl routing protocol for low power and lossy networks","volume":"16","author":"Kim","year":"2017","journal-title":"IEEE Trans. Mobile Comput."},{"key":"ref_21","first-page":"27","article-title":"SL-RPL: Stability-aware load balancing for RPL","volume":"13","author":"Wang","year":"2020","journal-title":"Trans. Mach. Learn. Data Min."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1038\/nmeth.4551","article-title":"Points of significance: Machine learning: Supervised methods","volume":"15","author":"Bzdok","year":"2018","journal-title":"Nat. Methods"},{"key":"ref_23","first-page":"1417","article-title":"Online linear regression and its application to model-based reinforcement learning","volume":"20","author":"Strehl","year":"2008","journal-title":"Adv. Neur. Inf. Process. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2861","DOI":"10.1016\/j.sigpro.2007.05.013","article-title":"Processing k nearest neighbor queries in location-aware sensor networks","volume":"12","author":"Xu","year":"2007","journal-title":"Signal Process."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Dong, Z., Zhao, Y., and Chen, Z. (2018, January 6\u201310). Support vector machine for channel prediction in high-speed railway communication systems. Proceedings of the IEEE MTT-S International Wireless Symposium (IWS), Chengdu, China.","DOI":"10.1109\/IEEE-IWS.2018.8400912"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1356","DOI":"10.1109\/TWC.2014.2365813","article-title":"Channel Estimation for Massive MIMO Using Gaussian-Mixture Bayesian Learning","volume":"14","author":"Wen","year":"2015","journal-title":"IEEE Trans. Wireless Commun."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/MSP.2018.2825478","article-title":"IoT Security Techniques Based on Machine Learning: How Do IoT Devices Use AI to Enhance Security?","volume":"35","author":"Xiao","year":"2018","journal-title":"IEEE Signal. Process. Mag."},{"key":"ref_28","first-page":"1","article-title":"The limitations in the state-of-the-art counter-measures against the security threats in H-IoT","volume":"2020","author":"Qadri","year":"2020","journal-title":"Cluster Comput."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Slivkins, A. (2019). Introduction to multi-armed bandits. arXiv.","DOI":"10.1561\/9781680836219"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Tokic, M. (2010, January 21\u201324). Adaptive s-greedy exploration in reinforcement learning based on value differences. Proceedings of the 33rd annual German Conference on Advances in Artificial Intelligence, Karlsruhe, Germany.","DOI":"10.1007\/978-3-642-16111-7_23"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.comcom.2017.08.005","article-title":"A reinforcement learning-based link quality estimation strategy for RPL and its impact on topology management","volume":"112","author":"Ancillotti","year":"2017","journal-title":"Comput. Commun."},{"key":"ref_32","unstructured":"Aziz, M. (2019). On Multi-Armed Bandits Theory and Applications. [Ph.D. Thesis, Northeastern University]."},{"key":"ref_33","first-page":"1","article-title":"Toward improved RPL: A congestion avoidance multipath routing protocol with time factor for wireless sensor networks","volume":"2016","author":"Tang","year":"2016","journal-title":"J. Sens."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Bhandari, K., Hosen, A., and Cho, G. (2018). CoAR: Congestion-aware routing protocol for low power and lossy networks for IoT applications. Sensors, 18.","DOI":"10.3390\/s18113838"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zier, A., Abouaissa, A., and Lorenz, P. (2018, January 9\u201313). E-RPL: A Routing Protocol for IoT Networks. Proceedings of the IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, UAE.","DOI":"10.1109\/GLOCOM.2018.8647318"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Fabian, P., Rachedi, A., Gueguen, C., and Lohier, S. (2018, January 9\u201313). Fuzzy-based Objective Function for Routing Protocol in the Internet of Things. Proceedings of the IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, UAE.","DOI":"10.1109\/GLOCOM.2018.8647969"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ghaleb, B., Al-Dubai, A., Ekonomou, E., Gharib, W., Mackenzi, L., and Bani Khala, M. (2018, January 28\u201330). A New Load-Balancing Aware Objective Function for RPL\u2019s IoT Networks. Proceedings of the IEEE 20th International Conference on High Performance Computing and Communications, IEEE 16th International Conference on Smart City, IEEE 4th International Conference on Data Science and Systems (HPCC\/SmartCity\/DSS), Exeter, UK.","DOI":"10.1109\/HPCC\/SmartCity\/DSS.2018.00151"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Cao, Y., and Wu, M. (2018). A novel RPL algorithm based on chaotic genetic algorithm. Sensors, 18.","DOI":"10.3390\/s18113647"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"23277","DOI":"10.1109\/ACCESS.2018.2817128","article-title":"CLRPL: Context-Aware and Load Balancing RPL for Iot Networks Under Heavy and Highly Dynamic Load","volume":"6","author":"Taghizadeh","year":"2018","journal-title":"IEEE Access"},{"key":"ref_40","unstructured":"Adams, J.T. (2006, January 4\u201311). An introduction to IEEE STD 802.15.4. Proceedings of the IEEE Aerospace Conference, Big Sky, MT, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1049\/el.2018.0617","article-title":"Channel observation-based scaled backoff mechanism for high-efficiency WLANs","volume":"54","author":"Ali","year":"2018","journal-title":"Electron. Lett."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1080\/02564602.2016.1242387","article-title":"Design of MAC Layer Resource Allocation Schemes for IEEE 802.11ax: Future Directions","volume":"35","author":"Ali","year":"2016","journal-title":"IETE Tech. Rev."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ali, R., Shahin, N., Musaddiq, A., Kim, B., and Kim, S.W. (2018, January 3\u20136). Fair and Efficient Channel Observation-Based Listen-Before Talk (CoLBT) for LAA-WiFi Coexistence in Unlicensed LTE. Proceedings of the 2018 Tenth International Conference on Ubiquitous and Future Networks (ICUFN), Prague, Czech Republic.","DOI":"10.1109\/ICUFN.2018.8436776"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Levis, P., Clausen, T., Hui, J., Gnawali, O., and Ko, J. (2011). The Trickle Algorithm, Internet Engineering Task Force (IETF). RFC 6206.","DOI":"10.17487\/rfc6206"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Musaddiq, A., Zikria, Y.B., and Kim, S.W. (2018, January 21\u201323). Energy-Aware Adaptive Trickle Timer Algorithm for RPL-based Routing in the Internet of Things. Proceedings of the 28th International Telecommunication Networks and Applications Conference (ITNAC), Sydney, NSW, Australia.","DOI":"10.1109\/ATNAC.2018.8615408"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Van Otterlo, M., and Wiering, M. (2012). Reinforcement learning and Markov decision processes. Reinforcement Learning, Springer.","DOI":"10.1007\/978-3-642-27645-3_1"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Narten, T., Nordmark, E., Simpson, W., and Soliman, H. (2007). Neighbor Discovery for IP version 6 (IPv6), Internet Engineering Task Force (IETF). RFC 4861.","DOI":"10.17487\/rfc4861"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1023\/A:1016637622896","article-title":"A capacity analysis for the IEEE 802.11 MAC protocol","volume":"7","author":"Tay","year":"2001","journal-title":"Wireless Netw."},{"key":"ref_49","unstructured":"Vu, H.L., and Sakurai, T. (2006, January 4\u20136). Collision probability in saturated IEEE 802.11 networks. Proceedings of the Australian Telecommunication Networks and Applications Conference, Melbourne, Australia."},{"key":"ref_50","first-page":"1","article-title":"Hidden node aware routing method using high-sensitive sensing device for multi-hop wireless mesh network","volume":"114","author":"Parvin","year":"2011","journal-title":"J. Wireless Commun. Netw."},{"key":"ref_51","unstructured":"(2020, May 03). Contiki: The Open Source Operating System for the Internet of Things. Available online: http:\/\/www.contiki-os.org\/."},{"key":"ref_52","unstructured":"(2020, May 03). Z1 Datasheet. Available online: http:\/\/zolertia.sourceforge.net\/wiki\/images\/e\/e8\/Z1_RevC_Datasheet.pdf."},{"key":"ref_53","unstructured":"(2020, May 03). Contiki Netstack. Available online: http:\/\/anrg.usc.edu\/contiki\/index.php\/Network_Stack."},{"key":"ref_54","unstructured":"Melo, F.S. (2001). Convergence of Q-Learning: A simple Proof, Institute of Systems and Robotics. Institute Of Systems and Robotics, Tech. Rep (2001)."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/15\/4158\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:51:53Z","timestamp":1760176313000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/15\/4158"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,26]]},"references-count":54,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2020,8]]}},"alternative-id":["s20154158"],"URL":"https:\/\/doi.org\/10.3390\/s20154158","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,26]]}}}