{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T16:37:07Z","timestamp":1778603827075,"version":"3.51.4"},"reference-count":36,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T00:00:00Z","timestamp":1665360000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["U19B2016"],"award-info":[{"award-number":["U19B2016"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>With the development of artificial intelligence, intelligent communication jamming decision making is an important research direction of cognitive electronic warfare. In this paper, we consider a complex intelligent jamming decision scenario in which both communication parties choose to adjust physical layer parameters to avoid jamming in a non-cooperative scenario and the jammer achieves accurate jamming by interacting with the environment. However, when the situation becomes complex and large in number, traditional reinforcement learning suffers from the problems of failure to converge and a high number of interactions, which are fatal and unrealistic in a real warfare environment. To solve this problem, we propose a deep reinforcement learning based and maximum-entropy-based soft actor-critic (SAC) algorithm. In the proposed algorithm, we add an improved Wolpertinger architecture to the original SAC algorithm in order to reduce the number of interactions and improve the accuracy of the algorithm. The results show that the proposed algorithm shows excellent performance in various scenarios of jamming and achieves accurate, fast, and continuous jamming for both sides of the communication.<\/jats:p>","DOI":"10.3390\/e24101441","type":"journal-article","created":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T21:07:11Z","timestamp":1665436031000},"page":"1441","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Deep Reinforcement Learning Based Decision Making for Complex Jamming Waveforms"],"prefix":"10.3390","volume":"24","author":[{"given":"Yuting","family":"Xu","sequence":"first","affiliation":[{"name":"Key Laboratory of RF Circuits and Systems, Ministry of Education, Hangzhou Dianzi University, Hangzhou 310018, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chao","family":"Wang","sequence":"additional","affiliation":[{"name":"Key Laboratory of RF Circuits and Systems, Ministry of Education, Hangzhou Dianzi University, Hangzhou 310018, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5003-7325","authenticated-orcid":false,"given":"Jiakai","family":"Liang","sequence":"additional","affiliation":[{"name":"Key Laboratory of RF Circuits and Systems, Ministry of Education, Hangzhou Dianzi University, Hangzhou 310018, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0880-9798","authenticated-orcid":false,"given":"Keqiang","family":"Yue","sequence":"additional","affiliation":[{"name":"Key Laboratory of RF Circuits and Systems, Ministry of Education, Hangzhou Dianzi University, Hangzhou 310018, China"},{"name":"Science and Technology on Communication Information Security Control Laboratory, The No. 011 Research Center, Jiaxing 314033, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenjun","family":"Li","sequence":"additional","affiliation":[{"name":"Key Laboratory of RF Circuits and Systems, Ministry of Education, Hangzhou Dianzi University, Hangzhou 310018, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3089-4346","authenticated-orcid":false,"given":"Shilian","family":"Zheng","sequence":"additional","affiliation":[{"name":"Science and Technology on Communication Information Security Control Laboratory, The No. 011 Research Center, Jiaxing 314033, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhijin","family":"Zhao","sequence":"additional","affiliation":[{"name":"The School of Communication Engineering, Hangzhou Dianzi University, Hangzhou 310018, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2792","DOI":"10.1109\/TWC.2015.2510643","article-title":"Jamming Bandits-A Novel Learning Method for Optimal Jamming","volume":"15","author":"Amuru","year":"2016","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1474","DOI":"10.1109\/TVT.2021.3053015","article-title":"Reinforcement learning assisted impersonation attack detection in device-to-device communications","volume":"70","author":"Tu","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3768","DOI":"10.1109\/TIT.2016.2551225","article-title":"Frequency hopping sequences with optimal partial Hamming correlation","volume":"62","author":"Bao","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Sun, J., and Li, X. (2016, January 6\u201310). Carrier frequency offset synchronization algorithm for short burst communication system. Proceedings of the 2016 IEEE 13th International Conference on Signal Processing (ICSP), Chengdu, China.","DOI":"10.1109\/ICSP.2016.7878023"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1109\/98.788210","article-title":"Cognitive radio: Making software radios more personal","volume":"6","author":"Mitola","year":"1999","journal-title":"IEEE Pers. Commun."},{"key":"ref_6","unstructured":"Karhima, T., Silvennoinen, A., Hall, M., and Haggman, S.-G. (November, January 31). IEEE 802.11 b\/g WLAN tolerance to jamming. Proceedings of the IEEE MILCOM 2004 Military Communications Conference, Monterey, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1504\/IJAHUC.2014.066419","article-title":"Jamming and anti-jamming techniques in wireless networks: A survey","volume":"17","author":"Grover","year":"2014","journal-title":"Int. J. Ad Hoc Ubiquitous Comput."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1109\/SURV.2011.041110.00022","article-title":"Denial of service attacks in wireless networks: The case of jammers","volume":"13","author":"Pelechrinis","year":"2010","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3799","DOI":"10.1016\/j.comnet.2013.09.004","article-title":"Joint reactive jammer detection and localization in an enterprise WiFi network","volume":"57","author":"Cai","year":"2013","journal-title":"Comput. Netw."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Schulz, M., Gringoli, F., Steinmetzer, D., Koch, M., and Hollick, M. (2017, January 18\u201320). Massive reactive smartphone-based jamming using arbitrary waveforms and adaptive power control. Proceedings of the 10th ACM Conference on Security and Privacy in Wireless and Mobile Networks, Boston, MA, USA.","DOI":"10.1145\/3098243.3098253"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/MCOM.2016.7452266","article-title":"LTE\/LTE-A jamming, spoofing, and sniffing: Threat assessment and mitigation","volume":"54","author":"Lichtman","year":"2016","journal-title":"IEEE Commun. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Broustis, I., Pelechrinis, K., Syrivelis, D., Krishnamurthy, S.V., and Tassiulas, L. (2009, January 14\u201318). FIJI: Fighting implicit jamming in 802.11 WLANs. Proceedings of the International Conference on Security and Privacy in Communication Systems, Athens, Greece.","DOI":"10.1007\/978-3-642-05284-2_2"},{"key":"ref_13","first-page":"5","article-title":"Investigating the effect of jamming attacks on wireless LANS","volume":"99","author":"Bandaru","year":"2014","journal-title":"Int. J. Comput. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1109\/COMST.2022.3159185","article-title":"Jamming attacks and anti-jamming strategies in wireless networks: A comprehensive survey","volume":"24","author":"Pirayesh","year":"2022","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zheng, C., Kasihmuddin, M.S.M., Mansor, M., Chen, J., and Guo, Y. (2022). Intelligent Multi-Strategy Hybrid Fuzzy K-Nearest Neighbor Using Improved Hybrid Sine Cosine Algorithm. Mathematics, 10.","DOI":"10.3390\/math10183368"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Zamri, N.E., Mansor, M.A., Mohd Kasihmuddin, M.S., Alway, A., Mohd Jamaludin, S.Z., and Alzaeemi, S.A. (2020). Amazon employees resources access data extraction via clonal selection algorithm and logic mining approach. Entropy, 22.","DOI":"10.3390\/e22060596"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Muhammad Sidik, S.S., Zamri, N.E., Mohd Kasihmuddin, M.S., Wahab, H.A., Guo, Y., and Mansor, M.A. (2022). Non-Systematic Weighted Satisfiability in Discrete Hopfield Neural Network Using Binary Artificial Bee Colony Optimization. Mathematics, 10.","DOI":"10.3390\/math10071129"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhu, Q., Li, H., Han, Z., and Ba\u015far, T. (2010, January 23\u201327). A stochastic game model for jamming in multi-channel cognitive radio systems. Proceedings of the 2010 IEEE International Conference on Communications, Cape Town, South Africa.","DOI":"10.1109\/ICC.2010.5502451"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"7490895","DOI":"10.1155\/2018\/7490895","article-title":"Multiobjective cognitive cooperative jamming decision-making method based on Tabu search-artificial bee colony algorithm","volume":"2018","author":"Ye","year":"2018","journal-title":"Int. J. Aerosp. Eng."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, L., Tang, L., Zhang, S., Wang, Z., Shen, X., and Zhang, Z. (2021). A self-adaptive reinforcement-exploration Q-learning algorithm. Symmetry, 13.","DOI":"10.3390\/sym13061057"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3133","DOI":"10.1109\/COMST.2019.2916583","article-title":"Applications of deep reinforcement learning in communications and networking: A survey","volume":"21","author":"Luong","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Chen, T., Liu, J., Xiao, L., and Huang, L. (2015, January 9\u201312). Anti-jamming transmissions with learning in heterogenous cognitive radio networks. Proceedings of the 2015 IEEE Wireless Communications and Networking Conference Workshops (WCNCW), New Orleans, LA, USA.","DOI":"10.1109\/WCNCW.2015.7122570"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Das, A., Ghosh, S.C., Das, N., and Barman, A.D. (2017, January 9\u201312). Q-learning based co-operative spectrum mobility in cognitive radio networks. Proceedings of the 2017 IEEE 42nd Conference on Local Computer Networks (LCN), Singapore.","DOI":"10.1109\/LCN.2017.80"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1013689704352","article-title":"Finite-time analysis of the multiarmed bandit problem","volume":"47","author":"Auer","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/s13638-019-1414-4","article-title":"An algorithm for jamming strategy using OMP and MAB","volume":"2019","author":"ZhuanSun","year":"2019","journal-title":"EURASIP J. Wirel. Commun. Netw."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4655","DOI":"10.1109\/TIT.2007.909108","article-title":"Signal recovery from random measurements via orthogonal matching pursuit","volume":"53","author":"Tropp","year":"2007","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Li, Y., Wang, X., Liu, D., Guo, Q., Liu, X., Zhang, J., and Xu, Y. (2019). On the performance of deep reinforcement learning-based anti-jamming method confronting intelligent jammer. Appl. Sci., 9.","DOI":"10.3390\/app9071361"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"103002","DOI":"10.1016\/j.dsp.2021.103002","article-title":"Reinforcement learning based a non-zero-sum game for secure transmission against smart jamming","volume":"112","author":"Zhao","year":"2021","journal-title":"Digit. Signal Process."},{"key":"ref_30","unstructured":"Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P., Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., and Coppin, B. (2015). Deep reinforcement learning in large discrete action spaces. arXiv."},{"key":"ref_31","unstructured":"Chen, H., Dai, X., Cai, H., Zhang, W., Wang, X., Tang, R., Zhang, Y., and Yu, Y. (February, January 27). Large-scale interactive recommendation with tree-structured policy gradient. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_32","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden."},{"key":"ref_33","unstructured":"Richard, S.S., and Andrew, G.B. (1998). The Reinforcement Learning Problem. Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_34","unstructured":"Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., and Abbeel, P. (2018). Soft actor-critic algorithms and applications. arXiv."},{"key":"ref_35","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_36","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/10\/1441\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:49:27Z","timestamp":1760143767000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/10\/1441"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,10]]},"references-count":36,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["e24101441"],"URL":"https:\/\/doi.org\/10.3390\/e24101441","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,10]]}}}