{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T14:26:34Z","timestamp":1781360794945,"version":"3.54.1"},"reference-count":53,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2024,8,27]],"date-time":"2024-08-27T00:00:00Z","timestamp":1724716800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["62301581"],"award-info":[{"award-number":["62301581"]}]},{"name":"National Natural Science Foundation of China","award":["2023M734313"],"award-info":[{"award-number":["2023M734313"]}]},{"name":"National Natural Science Foundation of China","award":["CX20230045"],"award-info":[{"award-number":["CX20230045"]}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["62301581"],"award-info":[{"award-number":["62301581"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["2023M734313"],"award-info":[{"award-number":["2023M734313"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["CX20230045"],"award-info":[{"award-number":["CX20230045"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Postgraduate Scientific Research Innovation Project of Hunan Province","award":["62301581"],"award-info":[{"award-number":["62301581"]}]},{"name":"Postgraduate Scientific Research Innovation Project of Hunan Province","award":["2023M734313"],"award-info":[{"award-number":["2023M734313"]}]},{"name":"Postgraduate Scientific Research Innovation Project of Hunan Province","award":["CX20230045"],"award-info":[{"award-number":["CX20230045"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>With the advancement of radar technology toward multifunctionality and cognitive capabilities, traditional radar countermeasures are no longer sufficient to meet the demands of countering the advanced multifunctional radar (MFR) systems. Rapid and accurate generation of the optimal jamming strategy is one of the key technologies for efficiently completing radar countermeasures. To enhance the efficiency and accuracy of jamming policy generation, an efficient jamming policy generation method based on multi-timescale ensemble Q-learning (MTEQL) is proposed in this paper. First, the task of generating jamming strategies is framed as a Markov decision process (MDP) by constructing a countermeasure scenario between the jammer and radar, while analyzing the principle radar operation mode transitions. Then, multiple structure-dependent Markov environments are created based on the real-world adversarial interactions between jammers and radars. Q-learning algorithms are executed concurrently in these environments, and their results are merged through an adaptive weighting mechanism that utilizes the Jensen\u2013Shannon divergence (JSD). Ultimately, a low-complexity and near-optimal jamming policy is derived. Simulation results indicate that the proposed method has superior jamming policy generation performance compared with the Q-learning algorithm, in terms of the short jamming decision-making time and low average strategy error rate.<\/jats:p>","DOI":"10.3390\/rs16173158","type":"journal-article","created":{"date-parts":[[2024,8,27]],"date-time":"2024-08-27T03:51:06Z","timestamp":1724730666000},"page":"3158","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Efficient Jamming Policy Generation Method Based on Multi-Timescale Ensemble Q-Learning"],"prefix":"10.3390","volume":"16","author":[{"given":"Jialong","family":"Qian","sequence":"first","affiliation":[{"name":"College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qingsong","family":"Zhou","sequence":"additional","affiliation":[{"name":"College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhihui","family":"Li","sequence":"additional","affiliation":[{"name":"College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhongping","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shasha","family":"Shi","sequence":"additional","affiliation":[{"name":"College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhenjia","family":"Xu","sequence":"additional","affiliation":[{"name":"College of Electronic Engineering, National University of Defense Technology, Hefei 230037, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiyun","family":"Xu","sequence":"additional","affiliation":[{"name":"Unit 93216 of PLA, Beijing 100085, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,27]]},"reference":[{"key":"ref_1","unstructured":"Zhou, Y., An, W., Guo, F., Liu, Z., and Jiang, W. (2014). Principles and Technologies of Electronic Warfare System, Publishing House of Electronics Industry."},{"key":"ref_2","first-page":"1","article-title":"Overview of cognitive electronic warfare","volume":"45","author":"Huang","year":"2023","journal-title":"J. Natl. Univ. Def. Technol."},{"key":"ref_3","unstructured":"Charlish, A. (2011). Autonomous Agents for Multi-Function Radar Resource Management. [Ph.D. Thesis, University College London]."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Apfeld, S., Charlish, A., and Ascheid, G. (2019, January 16\u201319). Modelling, learning and prediction of complex radar emitter behaviour. Proceedings of the 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA), Boca Raton, FL, USA.","DOI":"10.1109\/ICMLA.2019.00057"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"11205","DOI":"10.1109\/JSEN.2019.2936659","article-title":"Adaptive scheduling algorithm based on cpi and impact of tasks for multifunction radar","volume":"19","author":"Liu","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"5060","DOI":"10.1109\/TAES.2024.3384950","article-title":"Syntactic modeling and neural based parsing for multifunction radar signal interpretation","volume":"60","author":"Han","year":"2024","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_7","first-page":"17","article-title":"Recognition, inference, and prediction of advanced Multi-Function radar systems behaviors: Overview and prospects","volume":"40","author":"Wang","year":"2024","journal-title":"J. Signal Process."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1109\/TAES.1978.308584","article-title":"Radar Electronic Counter-Countermeasures","volume":"AES14","author":"Johnston","year":"1978","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1603","DOI":"10.1360\/N112018-00153","article-title":"The architecture and technology of cognitive electronic warfare","volume":"48","author":"Wang","year":"2018","journal-title":"Sci. Sin. Inform."},{"key":"ref_10","first-page":"118","article-title":"EW 104: Electronic Warfare Against a New Generation of Threats","volume":"67","author":"Dahle","year":"2024","journal-title":"Microw. J."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/MSP.2006.1593335","article-title":"Cognitive radar: A way of the future","volume":"23","author":"Haykin","year":"2006","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_12","first-page":"56","article-title":"A Model-Free Cognitive Anti-Jamming Strategy Using Adversarial Learning Algorithm","volume":"22","author":"Sudha","year":"2022","journal-title":"Cybern. Inf. Technol."},{"key":"ref_13","unstructured":"Darpa, A. (2010). Behavioral Learning for Adaptive Electronic Warfare. Darpa-BAA-10-79, Defense Advanced Research Projects Agency."},{"key":"ref_14","first-page":"56","article-title":"Regaining the advantage\u2014Cognitive electronic warfare","volume":"39","author":"Knowles","year":"2016","journal-title":"J. Electron. Def."},{"key":"ref_15","unstructured":"Zhou, H. (2018, January 14\u201316). An introduction of cognitive electronic warfare system. Proceedings of the International Conferences on Communications, Signal Processing, and Systems, Dalian, China."},{"key":"ref_16","first-page":"3580536","article-title":"Modeling and simulation for the investigation of radar responses to electronic attacks in electronic warfare environments","volume":"2018","author":"So","year":"2018","journal-title":"Secur. Commun. Netw."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"224761","DOI":"10.1109\/ACCESS.2020.3044453","article-title":"Artificial Intelligence Aided Electronic Warfare Systems- Recent Trends and Evolving Applications","volume":"8","author":"Purabi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"012119","DOI":"10.1088\/1757-899X\/1047\/1\/012119","article-title":"Cognitive radar control system using machine learning","volume":"Volume 1047","author":"Nepryaev","year":"2021","journal-title":"IOP Conference Series: Materials Science and Engineering"},{"key":"ref_19","unstructured":"BIS Research (2021). Cognitive electronic warfare: An artificial intelligence approach. Microw. J., 64, 110."},{"key":"ref_20","unstructured":"du Plessis, W.P., and Osner, N.R. (2018, January 13\u201316). Cognitive electronic warfare (EW) systems as a training aid. Proceedings of the Electronic Warfare International Conference (EWCI), Bangalore, India."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"02015","DOI":"10.1051\/matecconf\/202235502015","article-title":"Modeling method of combat mission based on OODA loop","volume":"355","author":"Xiao","year":"2022","journal-title":"MATEC Web Conf."},{"key":"ref_22","first-page":"178","article-title":"Overview of jamming decision-making method for Multi-Function phased array radar","volume":"40","author":"Zhang","year":"2019","journal-title":"J. Ordnance Equip. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"11383","DOI":"10.1109\/JSEN.2023.3267068","article-title":"Radar jamming decision-making in cognitive electronic warfare: A review","volume":"23","author":"Zhang","year":"2023","journal-title":"IEEE Sens. J."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Liangliang, G., Shilong, W., and Tao, L. (2010, January 5\u20137). A radar emitter identification method based on pulse match template sequence. Proceedings of the 2010 2nd International Conference on Signal Processing Systems, Dalian, China.","DOI":"10.1109\/ICSPS.2010.5555410"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"72257","DOI":"10.1109\/ACCESS.2019.2920398","article-title":"Game theoretic strategies design for monostatic radar and jammer based on mutual information","volume":"7","author":"Li","year":"2019","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1081","DOI":"10.1109\/TAES.2011.5751244","article-title":"Game theoretic analysis of adaptive radar jamming","volume":"47","author":"Bachmann","year":"2011","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_27","first-page":"218","article-title":"Jamming design selection based on D-S Theory","volume":"202","author":"Sun","year":"2003","journal-title":"J. Proj. Rocket. Missiles Guid."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1054","DOI":"10.1109\/TNN.1998.712192","article-title":"Reinforcement learning: An introduction","volume":"19","author":"Sutton","year":"1998","journal-title":"Neural Netw. IEEE Trans."},{"key":"ref_29","first-page":"335","article-title":"On-line q-learning using connectionist systems","volume":"37","author":"Rummery","year":"1994","journal-title":"Tech. Rep."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1007\/BF00115009","article-title":"Learning to predict by the methods of temporal differences","volume":"3","author":"Sutton","year":"1988","journal-title":"Mach. Learn."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ribeiro, R., Koerich, A.L., and Enembreck, F. (2007, January 2\u20135). Noise tolerance in reinforcement learning algorithms. Proceedings of the 2007 IEEE\/WIC\/ACM International Conference on Intelligent Agent Technology (IAT\u201907), Fremont, CA, USA.","DOI":"10.1109\/IAT.2007.94"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Llorente, F., Martino, L., Read, J., and Delgado-G\u00f3mez, D. (2024). A survey of Monte Carlo methods for noisy and costly densities with application to reinforcement learning and ABC. Int. Stat. Rev., 1.","DOI":"10.1111\/insr.12573"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"15255","DOI":"10.1007\/s00521-024-09839-z","article-title":"Judgmentally adjusted Q-values based on Q-ensemble for offline reinforcement learning","volume":"36","author":"Liu","year":"2024","journal-title":"Neural Comput. Appl."},{"key":"ref_35","first-page":"1194","article-title":"Design of cognitive radar jamming based on Q-Learning Algorithm","volume":"35","author":"Li","year":"2015","journal-title":"Trans. Beijing Inst. Technol."},{"key":"ref_36","first-page":"1969","article-title":"Construction and key technologies of cognitive jamming decision-making system against MFR","volume":"42","author":"Zhang","year":"2020","journal-title":"Syst. Eng. Electron."},{"key":"ref_37","first-page":"2488","article-title":"Research on decision-making modeling of cognitive jamming for multi-functional radar based on Markov","volume":"44","author":"Zhu","year":"2022","journal-title":"Syst. Eng. Electron."},{"key":"ref_38","first-page":"52","article-title":"Multi-function radar jamming decision method based on planning steps adaptive Dyna-Q","volume":"41","author":"Zhu","year":"2022","journal-title":"Ordnance Ind. Autom."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"8647386","DOI":"10.1155\/2021\/8647386","article-title":"Cognitive electronic jamming decision-making method based on improved Q-learning algorithm","volume":"2021","author":"Li","year":"2021","journal-title":"Int. J. Aerosp. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zhang, C., Song, Y., Jiang, R., Hu, J., and Xu, S. (2023). A cognitive electronic jamming decision-making method based on q-learning and ant colony fusion algorithm. Remote Sens., 15.","DOI":"10.3390\/rs15123108"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zheng, S., Zhang, C., Hu, J., and Xu, S. (2024). Radar-jamming decision-making based on improved q-learning and fpga hardware implementation. Remote Sens., 16.","DOI":"10.3390\/rs16071190"},{"key":"ref_42","first-page":"819","article-title":"DQN based decision-making method of cognitive jamming against multifunctional radar","volume":"42","author":"Zhang","year":"2020","journal-title":"Syst. Eng. Electron."},{"key":"ref_43","first-page":"86","article-title":"Cognitive jamming decision-making method against multifunctional radar based on A3C","volume":"45","author":"Zou","year":"2023","journal-title":"Syst. Eng. Electron."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"112150","DOI":"10.1109\/ACCESS.2022.3214842","article-title":"Multifunctional radar cognitive jamming decision based on dueling double deep q-network","volume":"99","author":"Feng","year":"2022","journal-title":"IEEE Access"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"8782","DOI":"10.1109\/TAES.2023.3312231","article-title":"Jamming policy generation via heuristic programming reinforcement learning","volume":"59","author":"Zhang","year":"2023","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_46","unstructured":"Mao, S. (2021). Research on Intelligent Jamming Decision-Making Methods Based on Reinforcement Learning. [Master\u2019s Thesis, Harbin Institute of Technology]."},{"key":"ref_47","first-page":"75","article-title":"Multi-agent autonomous electronic jamming system based on information sharing","volume":"56","author":"Zhang","year":"2022","journal-title":"J. Zhejiang Univ. Eng. Sci."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"108965","DOI":"10.1109\/TAES.2023.3272307","article-title":"Joint optimization of jamming type selection and power control for countering multi-function radar based on deep reinforcement learning","volume":"59","author":"Pan","year":"2023","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"108965","DOI":"10.1016\/j.sigpro.2023.108965","article-title":"Performance analysis of deep reinforcement learning-based intelligent cooperative jamming method confronting multi-functional networked radar","volume":"207","author":"Zhang","year":"2023","journal-title":"Signal Process."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1126\/science.220.4598.671","article-title":"Optimization by simulated annealing","volume":"220","author":"Kirkpatrick","year":"1983","journal-title":"Science"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1063\/1.1699114","article-title":"Equation of state calculations by fast computing machines","volume":"21","author":"Metropolis","year":"1953","journal-title":"J. Chem. Phys."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1093\/biomet\/57.1.97","article-title":"Monte Carlo sampling methods using Markov chains and their applications","volume":"57","author":"Hastings","year":"1970","journal-title":"Biometrika"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1427","DOI":"10.1109\/TSP.2024.3372699","article-title":"Multi-timescale ensemble Q-learning for markov decision process policy optimization","volume":"72","author":"Bozkus","year":"2024","journal-title":"IEEE Trans. Signal Process."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/17\/3158\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:43:19Z","timestamp":1760110999000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/17\/3158"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,27]]},"references-count":53,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["rs16173158"],"URL":"https:\/\/doi.org\/10.3390\/rs16173158","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,27]]}}}