{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,7]],"date-time":"2025-11-07T09:27:00Z","timestamp":1762507620232,"version":"build-2065373602"},"reference-count":29,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2019,1,3]],"date-time":"2019-01-03T00:00:00Z","timestamp":1546473600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Transmission latency minimization and energy efficiency improvement are two main challenges in multi-hop Cognitive Radio Networks (CRN), where the knowledge of topology and spectrum statistics are hard to obtain. For this reason, a cross-layer routing protocol based on quasi-cooperative multi-agent learning is proposed in this study. Firstly, to jointly consider the end-to-end delay and power efficiency, a comprehensive utility function is designed to form a reasonable tradeoff between the two measures. Then the joint design problem is modeled as a Stochastic Game (SG), and a quasi-cooperative multi-agent learning scheme is presented to solve the SG, which only needs information exchange with previous nodes. To further enhance performance, experience replay is applied to the update of conjecture belief to break the correlations and reduce the variance of updates. Simulation results demonstrate that the proposed scheme is superior to traditional algorithms leading to a shorter delay, lower packet loss ratio and higher energy efficiency, which is close to the performance of an optimum scheme.<\/jats:p>","DOI":"10.3390\/s19010151","type":"journal-article","created":{"date-parts":[[2019,1,3]],"date-time":"2019-01-03T11:11:56Z","timestamp":1546513916000},"page":"151","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["A Cross-Layer Routing Protocol Based on Quasi-Cooperative Multi-Agent Learning for Multi-Hop Cognitive Radio Networks"],"prefix":"10.3390","volume":"19","author":[{"given":"Yihang","family":"Du","sequence":"first","affiliation":[{"name":"National University of Defense Technology, Shushan District, Hefei 230000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chun","family":"Chen","sequence":"additional","affiliation":[{"name":"Army Academy of Artillery and Air Defense, Shushan District, Hefei 230000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengfei","family":"Ma","sequence":"additional","affiliation":[{"name":"Army Academy of Artillery and Air Defense, Shushan District, Hefei 230000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Xue","sequence":"additional","affiliation":[{"name":"National University of Defense Technology, Shushan District, Hefei 230000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,1,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1530","DOI":"10.1002\/wcm.2443","article-title":"Evolution and Future Trends of Research in Cognitive Radio: A Contemporary Survey","volume":"15","author":"Hossain","year":"2015","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"888","DOI":"10.1109\/COMST.2015.2401597","article-title":"A Survey on Radio Resource Allocation in Cognitive Radio Sensor Networks","volume":"17","author":"Ahmad","year":"2015","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"10306","DOI":"10.3390\/s130810306","article-title":"A Subcarrier-Pair based Resource Allocation Scheme Using Proportional Fairness for Cooperative OFDM-based Cognitive Radio Networks","volume":"13","author":"Ma","year":"2013","journal-title":"Sensors"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Huang, J., Zeng, X., Jian, X., Tan, X., and Zhang, Q. (2017). Opportunistic Capacity-based Resource Allocation for Chunk-based Multi-carrier Cognitive Radio Sensor Networks. Sensors, 17.","DOI":"10.3390\/s17010175"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zareei, M., Islam, A.K.M.M., Baharun, S., Vargasrosales, C., Azpilicueta, L., and Mansoor, N. (2017). Medium Access Control Protocols for Cognitive Radio Ad Hoc Networks: A Survey. Sensors, 17.","DOI":"10.3390\/s17092136"},{"key":"ref_6","first-page":"96","article-title":"A survey on distributed channel selection technique using surf algorithm for information transfer in multi-hop cognitive radio networks","volume":"1","author":"Ruby","year":"2014","journal-title":"Int. Conf. Comput. Sci. Comput. Intell."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1958","DOI":"10.1109\/JSAC.2012.121111","article-title":"Spectrum-Aware Opportunistic Routing in Multi-Hop Cognitive Radio Networks","volume":"30","author":"Liu","year":"2012","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ding, L., Melodia, T., Batalama, S.N., and Matyjas, J.D. (2010, January 21\u201325). Distributed Routing, Relay Selection, and Spectrum Allocation in Cognitive and Cooperative Ad Hoc Networks. Proceedings of the 7th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks (SECON), Boston, MA, USA.","DOI":"10.1109\/SECON.2010.5508267"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Lai, L., Wang, J., Huang, A., and Shan, H. (2012, January 3\u20137). Routing and Resource Allocation with Collision Constraint in Multi-Hop Cognitive Radio Networks. Proceedings of the GLOBECOM Workshops, Anaheim, CA, USA.","DOI":"10.1109\/GLOCOMW.2012.6477709"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1109\/TNSM.2013.120413.120533","article-title":"An Economic Framework for Routing and Channel Allocation in Cognitive Wireless Mesh Networks","volume":"11","author":"Amini","year":"2014","journal-title":"IEEE Trans. Netw. Serv. Manag."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1109\/98.760423","article-title":"A Review of Current Routing Protocols for Ad Hoc Mobile Wireless Networks","volume":"6","author":"Royer","year":"2002","journal-title":"IEEE Pers. Commun."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1136","DOI":"10.1109\/SURV.2012.100412.00017","article-title":"A Survey on Machine-Learning Techniques in Cognitive Radios","volume":"15","author":"Bkassiny","year":"2013","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1109\/JSTSP.2018.2798920","article-title":"Spectrum Access in Cognitive Radio Using A Two Stage Reinforcement Learning Approach","volume":"12","author":"Raj","year":"2018","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Al-Rawi, H.A.A., Yau, K.L.A., and Mohamad, H. (2014, January 20\u201322). A Reinforcement Learning-based Routing Scheme for Cognitive Radio Ad Hoc Networks. Proceedings of the 7th IFIP Wireless and Mobile Networking Conference (WMNC), Vilamoura, Portugal.","DOI":"10.1109\/WMNC.2014.6878881"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2155","DOI":"10.1109\/TMC.2012.178","article-title":"Stochastic Power Adaptation with Multiagent Reinforcement Learning for Cognitive Wireless Mesh Networks","volume":"12","author":"Chen","year":"2013","journal-title":"IEEE Trans. Mobile Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.comcom.2017.02.013","article-title":"Non-Cooperative Reinforcement Learning based Routing in Cognitive Radio Networks","volume":"106","author":"Pourpeighambar","year":"2017","journal-title":"Comput. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Du, Y., Zhang, F., and Xue, L. (2018). A Kind of Joint Routing and Resource Allocation Scheme based on Prioritized Memories-Deep Q Network for Cognitive Radio Ad Hoc Networks. Sensors, 18.","DOI":"10.3390\/s18072119"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"El-Sherif, A.A., Mohamed, A., and Hu, Y.C. (2011, January 5\u20139). Joint Routing and Resource Allocation for Delay Sensitive Traffic in Cognitive Mesh Networks. Proceedings of the IEEE Globecom Workshops, Houston, TX, USA.","DOI":"10.1109\/GLOCOMW.2011.6162596"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Singh, K., and Moh, S. (2017). An Energy-Efficient and Robust Multipath Routing Protocol for Cognitive Radio Ad Hoc Networks. Sensors, 17.","DOI":"10.3390\/s17092027"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Al-Rawi, H.A.A., and Yau, K.L.A. (2013, January 16\u201319). Route Selection for Minimizing Interference to Primary Users in Cognitive Radio Networks: A Reinforcement Learning Approach. Proceedings of the IEEE Symposium on Computational Intelligence for Communication Systems and Networks (CIComms), Singapore.","DOI":"10.1109\/CICommS.2013.6582850"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wellens, M., Riihijarvi, J., and Mahonen, P. (2010, January 6\u20139). Evaluation of Adaptive MAC-Layer Sensing in Realistic Spectrum Occupancy Scenarios. Proceedings of the IEEE Symposium on New Frontiers in Dynamic Spectrum, Singapore.","DOI":"10.1109\/DYSPAN.2010.5457888"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3360","DOI":"10.1109\/TWC.2013.061713.121545","article-title":"Wireless Mesh Network in Smart Grid: Modeling and Analysis for Time Critical Communications","volume":"12","author":"Xu","year":"2013","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"4470","DOI":"10.1109\/TWC.2014.2314121","article-title":"Qos-Oriented Wireless Routing for Smart Meter Data Collection: Stochastic Learning on Graph","volume":"13","author":"Cao","year":"2014","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1109\/TWC.2013.112513.122082","article-title":"Joint Routing and Resource Allocation for Delay Minimization in Cognitive Radio based Mesh Networks","volume":"13","author":"Mohamed","year":"2014","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1080\/16843703.2008.11673177","article-title":"Analysis of the Discrete Time Geo\/Geo\/1 Queue with Single Working Vacation","volume":"5","author":"Li","year":"2016","journal-title":"Qual. Technol. Quantit. Manag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"095001","DOI":"10.1088\/1751-8121\/aaa883","article-title":"Mixed-Strategy Nash Equilibrium for A Discontinuous Symmetric N-Player Game","volume":"51","author":"Hilhorst","year":"2017","journal-title":"J. Phys. Math. Gen."},{"key":"ref_27","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (arXiv, 2013). Playing Atari with Deep Reinforcement Learning, arXiv."},{"key":"ref_28","unstructured":"Littman, M.L. (1999). A Unified Analysis of Value-Function-Based Reinforcement Learning Algorithms, MIT Press."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-Level Control Through Deep Reinforcement Learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/1\/151\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:23:25Z","timestamp":1760185405000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/1\/151"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,3]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,1]]}},"alternative-id":["s19010151"],"URL":"https:\/\/doi.org\/10.3390\/s19010151","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2019,1,3]]}}}