{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T13:51:01Z","timestamp":1762955461058,"version":"build-2065373602"},"reference-count":51,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2015,8,13]],"date-time":"2015-08-13T00:00:00Z","timestamp":1439424000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Malaysian Government Fundamental Research Grant(FRGS)","award":["03-01-14-1423FR"],"award-info":[{"award-number":["03-01-14-1423FR"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>It is well-known that clustering partitions network into logical groups of nodes in order to achieve energy efficiency and to enhance dynamic channel access in cognitive radio through cooperative sensing. While the topic of energy efficiency has been well investigated in conventional wireless sensor networks, the latter has not been extensively explored. In this paper, we propose a reinforcement learning-based spectrum-aware clustering algorithm that allows a member node to learn the energy and cooperative sensing costs for neighboring clusters to achieve an optimal solution. Each member node selects an optimal cluster that satisfies pairwise constraints, minimizes network energy consumption and enhances channel sensing performance through an exploration technique. We first model the network energy consumption and then determine the optimal number of clusters for the network. The problem of selecting an optimal cluster is formulated as a Markov Decision Process (MDP) in the algorithm and the obtained simulation results show convergence, learning and adaptability of the algorithm to dynamic environment towards achieving an optimal solution. Performance comparisons of our algorithm with the Groupwise Spectrum Aware (GWSA)-based algorithm in terms of Sum of Square Error (SSE), complexity, network energy consumption and probability of detection indicate improved performance from the proposed approach. The results further reveal that an energy savings of 9% and a significant Primary User (PU) detection improvement can be achieved with the proposed approach.<\/jats:p>","DOI":"10.3390\/s150819783","type":"journal-article","created":{"date-parts":[[2015,8,13]],"date-time":"2015-08-13T11:13:57Z","timestamp":1439464437000},"page":"19783-19818","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["An Energy-Efficient Spectrum-Aware Reinforcement  Learning-Based Clustering Algorithm for Cognitive Radio Sensor Networks"],"prefix":"10.3390","volume":"15","author":[{"given":"Ibrahim","family":"Mustapha","sequence":"first","affiliation":[{"name":"Department of Computer and Communications Systems Engineering and Wireless and Photonics Research Centre, Faculty of Engineering, Universiti Putra Malaysia, 43400 Serdang Selangor, Malaysia"},{"name":"Department of Electrical and Electronics Engineering, Faculty of Engineering, University of Maiduguri, P. M. B. 1069, Maiduguri, Nigeria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Borhanuddin","family":"Ali","sequence":"additional","affiliation":[{"name":"Department of Computer and Communications Systems Engineering and Wireless and Photonics Research Centre, Faculty of Engineering, Universiti Putra Malaysia, 43400 Serdang Selangor, Malaysia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohd","family":"Rasid","sequence":"additional","affiliation":[{"name":"Department of Computer and Communications Systems Engineering and Wireless and Photonics Research Centre, Faculty of Engineering, Universiti Putra Malaysia, 43400 Serdang Selangor, Malaysia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aduwati","family":"Sali","sequence":"additional","affiliation":[{"name":"Department of Computer and Communications Systems Engineering and Wireless and Photonics Research Centre, Faculty of Engineering, Universiti Putra Malaysia, 43400 Serdang Selangor, Malaysia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hafizal","family":"Mohamad","sequence":"additional","affiliation":[{"name":"Wireless Networks and Protocol Research Lab, MIMOS Berhad, Technology Park Malaysia,  57000 Kuala Lumpur, Malaysia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2015,8,13]]},"reference":[{"key":"ref_1","unstructured":"Federal Communications Commission (2003). Docket No 03-222 Notice of Proposed Rule Making and Order."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.neucom.2011.07.027","article-title":"Reinforcement learning based sensing policy optimization for energy efficient cognitive radio networks","volume":"80","author":"Oksanen","year":"2012","journal-title":"Neurocomputing"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1007\/s11276-012-0530-4","article-title":"Reinforcement learning for cooperative sensing gain in cognitive radio ad hoc networks","volume":"19","author":"Lo","year":"2013","journal-title":"Wirel. Netw."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1109\/LCOMM.2011.103111.111884","article-title":"Cooperative spectrum sensing in multiple antenna based cognitive radio network using an improved energy detector","volume":"16","author":"Singh","year":"2012","journal-title":"IEEE Commun. Lett."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/MNET.2009.5191144","article-title":"Cognitive radio sensor networks","volume":"23","author":"Akan","year":"2009","journal-title":"IEEE Netw."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2257","DOI":"10.1109\/JSEN.2014.2309138","article-title":"Scalable dynamic routing protocol for cognitive radio sensor networks","volume":"14","author":"Spachos","year":"2014","journal-title":"IEEE Sens. J."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Yau, K.L.A., Komisarczuk, P., and Teal, P.D. (2009, January 20\u201323). Cognitive radio-based wireless sensor networks: Conceptual design and open issues. Proceedings of the IEEE 34th Conference on Local Computer Networks, Zurich, Switzerland.","DOI":"10.1109\/LCN.2009.5355016"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"11196","DOI":"10.3390\/s130911196","article-title":"Cognitive radio wireless sensor networks: Applications, challenges and research trends","volume":"13","author":"Joshi","year":"2013","journal-title":"Sensors"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Munir, A., and Gordon-Ross, A. (2010). Optimization Approaches in Wireless Sensor Networks, InTech Open Access Publisher.","DOI":"10.5772\/13093"},{"key":"ref_10","first-page":"1","article-title":"A cluster-based selective cooperative spectrum sensing scheme in cognitive radio","volume":"2013","author":"Koo","year":"2013","journal-title":"EURASIP J. Wirel. Commun. Netw."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"11307","DOI":"10.3390\/s120811307","article-title":"A self-optimizing scheme for energy balanced routing in wireless sensor networks using sensorant","volume":"12","author":"Saleh","year":"2012","journal-title":"Sensors"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1077","DOI":"10.1007\/s11277-013-1056-5","article-title":"A review on hierarchical routing protocols for wireless sensor networks","volume":"72","author":"Manap","year":"2013","journal-title":"Wirel. Pers. Commun."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.jnca.2014.07.020","article-title":"Clustering algorithms for cognitive radio networks: A survey","volume":"45","author":"Yau","year":"2014","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ozger, M., and Akan, O.B. (2013, January 14\u201319). Event-driven spectrum-aware clustering in cognitive radio sensor networks. Proceedings of the IEEE 2013 INFOCOM, Turin, Italy.","DOI":"10.1109\/INFCOM.2013.6566943"},{"key":"ref_15","unstructured":"Zhao, J., Zheng, H., and Yang, G.-H. (2005, January 8\u201311). Distributed coordination in dynamic spectrum allocation networks. Proceedings of the 2005 First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, DySPAN 2005, Baltimore, MD, USA."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Pefkianakis, I., Wong, S.H., and Lu, S. (2008, January 14\u201317). Samer: Spectrum aware mesh routing in cognitive radio networks. Proceedings of the 3rd IEEE Symposium on New Frontiers in Dynamic Spectrum Access Networks, DySPAN 2008, Chicago, IL, USA.","DOI":"10.1109\/DYSPAN.2008.90"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3731","DOI":"10.1007\/s11434-012-5254-4","article-title":"Energy-efficient spectrum-aware clustering for cognitive radio sensor networks","volume":"57","author":"Zhang","year":"2012","journal-title":"Chin. Sci. Bull."},{"key":"ref_18","unstructured":"Zhang, H., Zhang, Z., Dai, H., Yin, R., and Chen, X. (2011, January 5\u20139). Distributed spectrum-aware clustering in cognitive radio sensor networks. Proceedings of the 2011 IEEE Global Telecommunications Conference (GLOBECOM 2011), Houston, TX, USA."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lo, B.F., and Akyildiz, I.F. (2010, January 26\u201330). Reinforcement learning-based cooperative sensing in cognitive radio ad hoc networks. Proceedings of the 2010 IEEE 21st International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC), Istanbul, Turkey.","DOI":"10.1109\/PIMRC.2010.5671686"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Faganello, L.R., Kunst, R., Both, C.B., Granville, L.Z., and Rochol, J. (2013, January 7\u201310). Improving reinforcement learning algorithms for dynamic spectrum allocation in cognitive sensor networks. Proceedings of the 2013 IEEE Wireless Communications and Networking Conference (WCNC), Shanghai, China.","DOI":"10.1109\/WCNC.2013.6554535"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Abolarinwa, J., Latiff, N., and Yusof, S. (2013, January 16\u201317). Channel access framework for cognitive radio-based wireless sensor networks using reinforcement learning. Proceedings of the 2013 IEEE Student Conference on Research and Development (SCOReD), Pulau Pinang, Malaysia.","DOI":"10.1109\/SCOReD.2013.7002615"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Panahi, F.H., and Ohtsuki, T. (2013, January 9\u201313). Optimal channel-sensing policy based on fuzzy q-learning process over cognitive radio systems. Proceedings of the 2013 IEEE International Conference on Communications (ICC), Budapest, Hungary.","DOI":"10.1109\/ICC.2013.6654941"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yau, K.-L., Komisarczuk, P., and Teal, P.D. (2010, January 23\u201327). Applications of reinforcement learning to cognitive radio networks. Proceedings of the 2010 IEEE International Conference on Communications Workshops (ICC), Cape Town, South Africa.","DOI":"10.1109\/ICCW.2010.5503970"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1109\/CC.2014.6969800","article-title":"Predictive spectrum sensing strategy based on reinforcement learning","volume":"11","author":"Qu","year":"2014","journal-title":"IEEE Commun. China"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yau, K.-L. (2012, January 8\u201310). Reinforcement learning approach for centralized cognitive radio systems. Proceedings of the IET International Conference on Wireless Communications and Applications (ICWCA 2012), Kuala Lumpur, Malaysia.","DOI":"10.1049\/cp.2012.2076"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Mendes, A.C., Augusto, C.H.P., da Silva, M.W.R., Guedes, R.M., and de Rezende, J.F. (2011, January 4\u20137). Channel sensing order for cognitive radio networks using reinforcement learning. Proceedings of the 2011 IEEE 36th Conference on Local Computer Networks (LCN), Bonn, Germany.","DOI":"10.1109\/LCN.2011.6115516"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1016\/S1474-6670(17)38315-5"},{"key":"ref_28","unstructured":"Gosavi, A. A tutorial for reinforcement learning. Available online: http:\/\/web.mst.edu\/~gosavia\/tutorial.pdf."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/BF00115009","article-title":"Learning to predict by the methods of temporal differences","volume":"3","author":"Sutton","year":"1988","journal-title":"Mach. Learn."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Chen, Z., and Qiu, R.C. (2011, January 17\u201320). Cooperative spectrum sensing using q-learning with experimental validation. Proceedings of the IEEE SoutheastCon, Nashville, TN, USA.","DOI":"10.1109\/SECON.2011.5752975"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zheng, K., and Li, H. (2010, January 6\u201310). Achieving energy efficiency via drowsy transmission in cognitive radio. Proceedings of the 2010 IEEE Global Telecommunications Conference (GLOBECOM 2010), Miami, FL, USA.","DOI":"10.1109\/GLOCOM.2010.5683355"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1109\/TMC.2004.41","article-title":"Heed: A hybrid, energy-efficient, distributed clustering approach for ad hoc sensor networks","volume":"3","author":"Younis","year":"2004","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_33","unstructured":"Wagstaff, K., Cardie, C., Rogers, S., and Schr\u00f6dl, S. (July, January 28). Constrained k-means clustering with background knowledge. Proceedings of the Eighteenth International Conference on Machine Learning ICML, Williamstown, MA, USA."},{"key":"ref_34","unstructured":"Klein, D., Kamvar, S.D., and Manning, C.D. (2002). From Instance-Level Constraints to Space-Level Constraints: Making the Most of Prior Knowledge in Data Clustering, Stanford InfoLab Publication Server."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"5294","DOI":"10.1109\/TVT.2009.2028030","article-title":"Optimization of cooperative sensing in cognitive radio networks: A sensing-throughput tradeoff view","volume":"58","author":"Peh","year":"2009","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1109\/TWC.2012.021412.021512.111398","article-title":"Energy-Efficient packet size optimization for cognitive radio sensor networks","volume":"11","author":"Oto","year":"2012","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"906","DOI":"10.1109\/TVT.2011.2180549","article-title":"Energy-efficient spectrum sensing and access for cognitive radio networks","volume":"61","author":"Wang","year":"2012","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/j.phycom.2010.12.003","article-title":"Cooperative spectrum sensing in cognitive radio networks: A survey","volume":"4","author":"Akyildiz","year":"2011","journal-title":"Phys. Commun."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"4388","DOI":"10.3390\/s150204388","article-title":"A soft-hard combination-based cooperative spectrum sensing scheme for cognitive radio networks","volume":"15","author":"Do","year":"2015","journal-title":"Sensors"},{"key":"ref_40","unstructured":"Heinzelman, W.R., Chandrakasan, A., and Balakrishnan, H. (2000, January 4\u20137). Energy-efficient communication protocol for wireless microsensor networks. Proceedings of the IEEE 33rd Annual Hawaii International Conference on System Sciences, Maui, HI, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1109\/TWC.2002.804190","article-title":"An application-specific protocol architecture for wireless microsensor networks","volume":"1","author":"Heinzelman","year":"2002","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1109\/LCOMM.2003.820097","article-title":"On the energy-efficient organization and the lifetime of multi-hop sensor networks","volume":"7","author":"Zhu","year":"2003","journal-title":"IEEE Commun. Lett."},{"key":"ref_43","unstructured":"S\u0142abicki, M., Wojciechowski, B., and Surmacz, T. (2012). Computer Networks, Springer."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wang, Q., Hempstead, M., and Yang, W. A realistic power consumption model for wireless sensor network devices. Proceedings of the 2006 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, 2006. SECON\u201906.","DOI":"10.1109\/SAHCN.2006.288433"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"259","DOI":"10.2528\/PIERB08122303","article-title":"An estimation of sensor energy consumption","volume":"12","author":"Halgamuge","year":"2009","journal-title":"Prog. Electromagn. Res. B"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1109\/MSP.2002.1012351","article-title":"Energy-efficient dsps for wireless sensor networks","volume":"19","author":"Wang","year":"2002","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Mustapha, I., Ali, B.M., Sali, A., and Rasid, M.F. (2014, January 24\u201326). Energy-aware cluster based cooperative spectrum sensing for cognitive radio sensor networks. Proceedings of the IEEE International Symposium on Telecommunication Technologies, Langkawi Island, Malaysia.","DOI":"10.1109\/ISTT.2014.7238174"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Gosavi, A. (2008, January 7\u201310). On step sizes, stochastic shortest paths, and survival probabilities in reinforcement learning. Proceedings of the 40th Conference on Winter Simulation, Miami, FL, USA.","DOI":"10.1109\/WSC.2008.4736109"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Maleki, S., Pandharipande, A., and Leus, G. (2009, January 3\u20135). Energy-efficient spectrum sensing for cognitive sensor networks. Proceedings of the IEEE 35th Annual Conference on Industrial Electronics, IECON\u201909, Porto, Portugal.","DOI":"10.1109\/IECON.2009.5415258"},{"key":"ref_50","unstructured":"Heinzelman, W.R., Sinha, A., Wang, A., and Chandrakasan, A.P. (2000, January 5\u20139). Energy-scalable algorithms and protocols for wireless microsensor networks. Proceedings of 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP\u201900, Istanbul, Turkey."},{"key":"ref_51","unstructured":"Aksin, D., Gregori, S., and Maloberti, F. (2005, January 23\u201326). High-efficiency power amplifier for wireless sensor networks. Proceedings of the IEEE International Symposium on Circuits and Systems, ISCAS 2005, Kobe, Japan."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/8\/19783\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:50:38Z","timestamp":1760215838000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/8\/19783"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,8,13]]},"references-count":51,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2015,8]]}},"alternative-id":["s150819783"],"URL":"https:\/\/doi.org\/10.3390\/s150819783","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2015,8,13]]}}}