{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T17:43:36Z","timestamp":1784742216975,"version":"3.55.0"},"reference-count":41,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2022,9,14]],"date-time":"2022-09-14T00:00:00Z","timestamp":1663113600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Japan Society for the Promotion of Science","award":["20K11764"],"award-info":[{"award-number":["20K11764"]}]},{"name":"Japan Society for the Promotion of Science","award":["2022-22FA01"],"award-info":[{"award-number":["2022-22FA01"]}]},{"name":"ROIS NII Open Collaborative Research","award":["20K11764"],"award-info":[{"award-number":["20K11764"]}]},{"name":"ROIS NII Open Collaborative Research","award":["2022-22FA01"],"award-info":[{"award-number":["2022-22FA01"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Future-generation wireless networks should accommodate surging growth in mobile data traffic and support an increasingly high density of wireless devices. Consequently, as the demand for spectrum continues to skyrocket, a severe shortage of spectrum resources for wireless networks will reach unprecedented levels of challenge in the near future. To deal with the emerging spectrum-shortage problem, dynamic spectrum access techniques have attracted a great deal of attention in both academia and industry. By exploiting the cognitive radio techniques, secondary users (SUs) are capable of accessing the underutilized spectrum holes of the primary users (PUs) to increase the whole system\u2019s spectral efficiency with minimum interference violations. In this paper, we mathematically formulate the spectrum access problem for interweave cognitive radio networks, and propose a usage-aware deep reinforcement learning based scheme to solve it, which exploits the historical channel usage data to learn the time correlation and channel correlation of the PU channels. We evaluated the performance of the proposed approach by extensive simulations in both uncorrelated and correlated PU channel usage cases. The evaluation results validate the superiority of the proposed scheme in terms of channel access success probability and SU-PU interference probability, by comparing it with ideal results and existing methods.<\/jats:p>","DOI":"10.3390\/s22186949","type":"journal-article","created":{"date-parts":[[2022,9,14]],"date-time":"2022-09-14T23:16:36Z","timestamp":1663197396000},"page":"6949","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["A Usage Aware Dynamic Spectrum Access Scheme for Interweave Cognitive Radio Network by Exploiting Deep Reinforcement Learning"],"prefix":"10.3390","volume":"22","author":[{"given":"Xiaoyan","family":"Wang","sequence":"first","affiliation":[{"name":"Graduate School of Science and Engineering, Ibaraki University, Mito 310-8512, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuto","family":"Teraki","sequence":"additional","affiliation":[{"name":"IVIS Cooperation, Tokyo 113-0033, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masahiro","family":"Umehira","sequence":"additional","affiliation":[{"name":"Faculty of Science and Technology, Nanzan University, Nagoya 466-0824, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Science and Technology of China, Hefei 230052, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4364-8491","authenticated-orcid":false,"given":"Yusheng","family":"Ji","sequence":"additional","affiliation":[{"name":"Information Systems Architecture Research Division, National Institute of Informatics, Tokyo 101-8430, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1109\/TMC.2011.128","article-title":"Mining spectrum usage data: A large-scale spectrum measurement study","volume":"11","author":"Yin","year":"2012","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2658","DOI":"10.1109\/TVT.2016.2581179","article-title":"Auction based frameworks for secure communications in static and dynamic cognitive radio networks","volume":"66","author":"Wang","year":"2017","journal-title":"IEee Trans. Veh. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"34675","DOI":"10.1109\/ACCESS.2020.2974227","article-title":"An Efficient privacy preserving spectrum sharing framework for internet of things","volume":"8","author":"Wang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1109\/TNET.2020.2986410","article-title":"Dynamic exclusion zones for protecting primary users in database-driven spectrum sharing","volume":"28","author":"Bhattarai","year":"2020","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Barb, G., Alexa, F., and Otesteanu, M. (2021). Dynamic spectrum sharing for future LTE-NR networks. Sensors, 21.","DOI":"10.3390\/s21124215"},{"key":"ref_6","unstructured":"Mueck, M.D., Srikanteswara, S., and Badi, B. (2021, October 01). Spectrum Sharing: Licensed Shared Access (lsa) and Spectrum Access System (sas). Available online: http:\/\/www.intel.com\/content\/dam\/www\/public\/us\/en\/documents\/white-papers\/spectrum-sharing-lsasas-paper.pdf."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Chakraborty, A., and Das, S.R. (2014, January 2\u20135). Measurement-augmented spectrum databases for white space spectrum. Proceedings of the ACM CoNEXT, Sydney, NSW, Australia.","DOI":"10.1145\/2674005.2675001"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Akimoto, M., Wang, X., Umehira, M., and Ji, Y. (2019, January 24\u201327). Crowdsourced Radio environment mapping by exploiting machine learning. Proceedings of the 22nd International Symposium on Wireless Personal Multimedia Communications (WPMC), Lisbon, Portugal.","DOI":"10.1109\/WPMC48795.2019.9096108"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, X., Umehira, M., Han, B., Li, P., Gu, Y., and Wu, C. (2019, January 20\u201324). Online incentive mechanism for crowdsourced radio environment map construction. Proceedings of the IEEE International Conference on Communications (IEEE ICC 2019), Shanghai, China.","DOI":"10.1109\/ICC.2019.8761268"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wang, X., Umehira, M., Akimoto, M., Han, B., and Zhou, H. (2022). Green spectrum sharing framework in B5G era by exploiting crowdsensing. IEEE Trans. On Green Commun. Networking, Early Access.","DOI":"10.1109\/TGCN.2022.3186282"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Li, H. (2010). Multiagent q-learning for aloha-like spectrum access in cognitive radio systems. J. Wirel. Com. Netw., 2010.","DOI":"10.1109\/SDR.2010.5507919"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1109\/MVT.2012.2190188","article-title":"Impact of cognitive radio: Recognition and informed exploitation of grey spectrum opportunities","volume":"7","author":"Macaluso","year":"2012","journal-title":"IEEE Veh. Technol. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Roy, D., Mukherjee, T., Chatterjee, M., and Pasiliao, E. (2019, January 11\u201314). Primary user activity prediction in DSA networks using recurrent structures. Proceedings of the IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), Newark, NJ, USA.","DOI":"10.1109\/DySPAN.2019.8935716"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1109\/TCCN.2020.2973572","article-title":"Spectrum availability prediction for cognitive radio communications: A DCG approach","volume":"6","author":"Yu","year":"2020","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1109\/TCCN.2020.3048105","article-title":"Prediction and Modeling of spectrum occupancy for dynamic spectrum access systems","volume":"7","author":"Li","year":"2021","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.1109\/TMC.2016.2592917","article-title":"Channel Selection algorithm for cognitive radio networks with heavy-tailed idle times","volume":"16","author":"Sengottuvelan","year":"2017","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1155\/2021\/9985972","article-title":"Reinforcement learning for joint channel\/subframe selection of LTE in the unlicensed spectrum","volume":"2021","author":"Kishimoto","year":"2021","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Amrallah, A., Mohamed, E.M., Tran, G.K., and Sakaguchi, K. (2021). Enhanced dynamic spectrum access in UAV wireless networks for post-disaster area surveillance system: A Multi-player multi-armed bandit approach. Sensors, 21.","DOI":"10.3390\/s21237855"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Teraki, Y., Wang, X., Umehira, M., and Ji, Y. (2021, January 12\u201316). Deep reinforcement learning based usage aware spectrum access scheme. Proceedings of the 24th International Symposium on Wireless Personal Multimedia Communications (WPMC), Okayama, Japan.","DOI":"10.1109\/WPMC52694.2021.9700468"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1109\/T-WC.2009.071363","article-title":"Optimal selection of channel sensing order in cognitive radio","volume":"8","author":"Jiang","year":"2009","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1109\/TMC.2015.2412940","article-title":"Opportunistic channel selection by cognitive wireless nodes under imperfect observations and limited memory: A repeated game model","volume":"15","author":"Khan","year":"2016","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"5431","DOI":"10.1109\/T-WC.2008.071349","article-title":"On myopic sensing for multi-channel opportunistic access: Structure, optimality, and performance","volume":"7","author":"Zhao","year":"2008","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Dai, W., Gai, Y., and Krishnamachari, B. (2014, January 1\u20133). Online learning for multi-channel opportunistic access over unknown markovian channels. Proceedings of the IEEE SECON, Singapore.","DOI":"10.1109\/SAHCN.2014.6990328"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2402","DOI":"10.1109\/LCOMM.2019.2941717","article-title":"Spectrum Sensing across multiple service providers: A discounted thompson sampling method","volume":"23","author":"Zhou","year":"2019","journal-title":"IEEE Commun. Lett."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3148","DOI":"10.1109\/TVT.2010.2048766","article-title":"Opportunistic bandwidth sharing through reinforcement learning","volume":"59","author":"Venkatraman","year":"2010","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Syed, A.R., Yau, K.L.A., Mohamad, H., Ramli, N., and Hashim, W. (2014, January 3\u20135). Channel selection in multi-hop cognitive radio network using reinforcement learning: An experimental study. Proceedings of the ICFCNA, Kuala Lumpur, Malaysia.","DOI":"10.1049\/cp.2014.1402"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Nguyen, H.Q., Nguyen, B.T., Dong, T.Q., Ngo, D.T., and Nguyen, T.A. (2018, January 22\u201325). Deep q-learning with multiband sensing for dynamic spectrum access. Proceedings of the IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), Seoul, Korea.","DOI":"10.1109\/DySPAN.2018.8610402"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"4244","DOI":"10.1109\/TII.2021.3113949","article-title":"Reinforcement-Learning-based dynamic spectrum access for software-defined cognitive industrial internet of things","volume":"18","author":"Liu","year":"2022","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1109\/TCCN.2018.2809722","article-title":"Deep Reinforcement learning for dynamic multichannel access in wireless networks","volume":"4","author":"Wang","year":"2018","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1109\/TCCN.2019.2952909","article-title":"A deep actor-critic reinforcement learning framework for dynamic multichannel access","volume":"5","author":"Zhong","year":"2019","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1109\/TWC.2018.2879433","article-title":"Deep multi-user reinforcement learning for distributed dynamic spectrum access","volume":"18","author":"Naparstek","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"4494","DOI":"10.1109\/TWC.2020.2984227","article-title":"The Application of deep reinforcement learning to distributed spectrum access in dynamic heterogeneous environments with partial observations","volume":"19","author":"Xu","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/TCCN.2017.2669985","article-title":"Optimal channel sensing strategy for cognitive radio networks with heavy-tailed idle times","volume":"3","author":"Senthilmurugan","year":"2017","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1109\/JSTSP.2018.2798920","article-title":"Spectrum Access in cognitive radio using a two-stage reinforcement learning approach","volume":"12","author":"Raj","year":"2018","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Sheng, X., and Wang, S. (2020, January 7\u201311). Sensing-Transmission tradeoff for multimedia transmission in cognitive radio networks. Proceedings of the GLOBECOM 2020\u20132020 IEEE Global Communications Conference, Taipei, Taiwan.","DOI":"10.1109\/GLOBECOM42002.2020.9322250"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1938","DOI":"10.1109\/JIOT.2018.2872441","article-title":"Distributive dynamic spectrum access through deep reinforcement learning: A reservoir computing-based approach","volume":"6","author":"Chang","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2237","DOI":"10.1109\/JSAC.2013.131115","article-title":"Complexity of Spectrum activity and benefits of reinforcement learning for dynamic channel selection","volume":"31","author":"Macaluso","year":"2013","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Watkins, C.J.C.H., and Dayan, P. (1992). Q-learning. Machine Learning, Springer.","DOI":"10.1007\/BF00992698"},{"key":"ref_40","unstructured":"(2021). MATLAB and Statistics Toolbox Release 2021b, The MathWorks, Inc."},{"key":"ref_41","unstructured":"Python Software Foundation (2022, August 01). Python Language Reference, Version 2.7. Available online: http:\/\/www.python.org."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/18\/6949\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:31:14Z","timestamp":1760142674000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/18\/6949"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,14]]},"references-count":41,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["s22186949"],"URL":"https:\/\/doi.org\/10.3390\/s22186949","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,14]]}}}