{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T01:33:50Z","timestamp":1770773630401,"version":"3.50.0"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T00:00:00Z","timestamp":1699920000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T00:00:00Z","timestamp":1699920000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"the Deep Sea Observation Project","award":["2019JCJQZD02400"],"award-info":[{"award-number":["2019JCJQZD02400"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["EURASIP J. Adv. Signal Process."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The Doppler shift of low-speed targets is frequently disturbed by the reverberation Doppler spread clutter under the shallow sea. The clutter is generated by underwater scatterers, which increases the difficulty of Doppler estimation. To solve this problem, a reverberation target resolution function based on the Doppler spread clutter statistical model is proposed in this paper. Through the width of reverberation Doppler clutter, this function adjusts the waveform parameters by determining whether the target is discriminable. In addition, the reverberation Doppler spread clutter is time-spatial varying and affected by grazing angle, waves, wind speed, fish and other effects. Thus, the sonar waveform parameters need to be adjusted constantly. Therefore, this paper combines the cognitive sonar based on reinforcement learning with the reverberation target resolution function to evaluate different waveforms in different environments. Consequently, the sonar can adjust the waveform parameters in real-time and obtain the optimal waveform in different environments. Meanwhile, in this paper, the action selection strategy of Dyna-Q reinforcement learning is optimized, and the model-based maximum action selection Dyna-Q algorithm (Dyna-Q-Max-Action) is proposed. Compared with the traditional Dyna-Q and Q-learning algorithms, the proposed algorithm needs fewer episodes. Finally, numerical simulation verified the effectiveness of the proposed algorithm.<\/jats:p>","DOI":"10.1186\/s13634-023-01054-7","type":"journal-article","created":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T11:02:28Z","timestamp":1699959748000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Model-based optimal action selection for Dyna-Q reverberation suppression cognitive sonar"],"prefix":"10.1186","volume":"2023","author":[{"given":"Yubin","family":"Fu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaochuan","family":"Ma","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chao","family":"Feng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xingxuan","family":"Pei","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengzhuo","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,11,14]]},"reference":[{"key":"1054_CR1","doi-asserted-by":"crossref","unstructured":"Z. Hao\u00a0ke, M. Qu\u00a0li, L. Hai\u00a0lin, Study on robust space-time adaptive reverberation suppressing, in IEEE 10th International Conference on Signal Processing Proceedings, pp. 2407\u20132410 (2010)","DOI":"10.1109\/ICOSP.2010.5655206"},{"key":"1054_CR2","doi-asserted-by":"crossref","unstructured":"X. Cui, C. Chi, S. Li, Y. Li, H. Huang, Coprime pulse trains of frequency-modulated for suppressing reverberation, in OCEANS 2021: San Diego Porto, pp. 1\u20134 (2021)","DOI":"10.23919\/OCEANS44145.2021.9705661"},{"issue":"3","key":"1054_CR3","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1121\/1.2838249","volume":"123","author":"BW Choi","year":"2008","unstructured":"B.W. Choi, E.H. Bae, J.S. Kim, K.K. Lee, Improved prewhitening method for linear frequency modulation reverberation using dechirping transformation. J. Acoust. Soc. Am. 123(3), 21\u201325 (2008)","journal-title":"J. Acoust. Soc. Am."},{"issue":"6","key":"1054_CR4","doi-asserted-by":"publisher","first-page":"3433","DOI":"10.1121\/1.419585","volume":"102","author":"JN Maksym","year":"1997","unstructured":"J.N. Maksym, M. Sandys-Wunsch, Adaptive beamforming against reverberation for a three-sensor array. J. Acoust. Soc. Am. 102(6), 3433\u20133438 (1997)","journal-title":"J. Acoust. Soc. Am."},{"key":"1054_CR5","unstructured":"Y. Li, H. Huang, C. Zhang, S. Li, New schur-type-based pci algorithms for reverberation suppression in active sonar, in Proceedings. (ICASSP \u201905). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005, vol. 4, pp. 641\u20136444 (2005)"},{"key":"1054_CR6","doi-asserted-by":"crossref","unstructured":"Z.-Q. Wang, L. An, J.-R. Lu, Signal detection based on mathematical morphology in oceanic reverberation, in 2007 14th International Conference on Mechatronics and Machine Vision in Practice, pp. 8\u201312 (2007)","DOI":"10.1109\/MMVIP.2007.4430705"},{"issue":"1","key":"1054_CR7","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1109\/MSP.2006.1593335","volume":"23","author":"S Haykin","year":"2006","unstructured":"S. Haykin, Cognitive radar: a way of the future. IEEE Signal Process. Mag. 23(1), 30\u201340 (2006)","journal-title":"IEEE Signal Process. Mag."},{"key":"1054_CR8","doi-asserted-by":"crossref","unstructured":"L. Xiaohua, L. Yaan, L. Guancheng, Y. Jing, Research of the principle of cognitive sonar and beamforming simulation analysis, in 2011 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), pp. 1\u20135 (2011)","DOI":"10.1109\/ICSPCC.2011.6061602"},{"key":"1054_CR9","doi-asserted-by":"crossref","unstructured":"T. Claussen, V.D. Nguyen, Real-time cognitive sonar system with target-optimized adaptive signal processing through multi-layer data fusion, in 2015 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pp. 357\u2013361 (2015)","DOI":"10.1109\/MFI.2015.7295834"},{"key":"1054_CR10","doi-asserted-by":"crossref","unstructured":"X. Qing, D. Nie, G. Qiao, J. Tang, Dolphin bio-inspired transmitting waveform design for cognitive sonar and its performance analysis, in 2016 IEEE\/OES China Ocean Acoustics (COA), pp. 1\u20137 (2016)","DOI":"10.1109\/COA.2016.7535649"},{"issue":"6419","key":"1054_CR11","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1126\/science.aar6404","volume":"362","author":"D Silver","year":"2018","unstructured":"D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, D. Hassabis, A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419), 1140\u20131144 (2018)","journal-title":"Science"},{"key":"1054_CR12","unstructured":"M. Taylor, Teaching reinforcement learning with mario: An argument and case study, in Proceedings of the National Conference on Artificial Intelligence 2 (2011)"},{"key":"1054_CR13","doi-asserted-by":"crossref","unstructured":"J.E. Summers, J.M. Trader C.F. Gaumond, J.L. Chen, Deep reinforcement learning for cognitive sonar. J. Acoust. Soc. Am. 143(3-Supplement), 1716\u20131716 (2018)","DOI":"10.1121\/1.5035588"},{"key":"1054_CR14","doi-asserted-by":"crossref","unstructured":"J. Tucker, V. Chavali, K.E. Wage, J.K. Nelson, Multiple objective optimization for fully adaptive active sonar, in OCEANS 2022, Hampton Roads, pp. 1\u20139 (2022)","DOI":"10.1109\/OCEANS47191.2022.9977227"},{"issue":"5","key":"1054_CR15","doi-asserted-by":"publisher","first-page":"3053","DOI":"10.1121\/1.4756921","volume":"132","author":"TC Yang","year":"2012","unstructured":"T.C. Yang, J. Schindall, C.-F. Huang, J.-Y. Liu, Clutter reduction using Doppler sonar in a harbor environment. J. Acoust. Soc. Am. 132(5), 3053\u20133067 (2012)","journal-title":"J. Acoust. Soc. Am."},{"key":"1054_CR16","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"2016","unstructured":"R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA, 2016)"},{"issue":"5","key":"1054_CR17","doi-asserted-by":"publisher","first-page":"4566","DOI":"10.1109\/TAES.2022.3164394","volume":"58","author":"X He","year":"2022","unstructured":"X. He, Y. Xu, M. Liu, C. Hao, C. Hou, Adaptive estimation of k-distribution shape parameter based on fuzzy statistical normalization processing. IEEE Trans. Aerosp. Electron. Syst. 58(5), 4566\u20134577 (2022)","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"1054_CR18","doi-asserted-by":"crossref","unstructured":"P.C. Etter, C.H. Haas, D.V. Ramani, Evolving trends and challenges in applied underwater acoustic modeling, in OCEANS 2015 - MTS\/IEEE Washington, pp. 1\u201310 (2015)","DOI":"10.23919\/OCEANS.2015.7401806"},{"key":"1054_CR19","doi-asserted-by":"crossref","unstructured":"F. Cao, X. Zhang, J. Han, S. Lv, Experimental analysis of statistical property of low frequency reverberation envelope in shallow water, in 2021 OES China Ocean Acoustics (COA), pp. 534\u2013538 (2021)","DOI":"10.1109\/COA50123.2021.9520013"},{"issue":"5","key":"1054_CR20","doi-asserted-by":"publisher","first-page":"2523","DOI":"10.1121\/1.4897406","volume":"136","author":"JJ Murray","year":"2014","unstructured":"J.J. Murray, A theoretical model of linearly filtered reverberation for pulsed active sonar in shallow water. J. Acoust. Soc. Am. 136(5), 2523\u20132531 (2014)","journal-title":"J. Acoust. Soc. Am."},{"key":"1054_CR21","first-page":"943","volume":"43","author":"C Zhang","year":"2018","unstructured":"C. Zhang, X. Ma, X. Li, F. Zhan, S. Zhang, Modified asymmetric statistical model for the reverberation doppler spread spectrum. Shengxue Xuebao\/Acta Acustica 43, 943\u2013950 (2018)","journal-title":"Shengxue Xuebao\/Acta Acustica"},{"key":"1054_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13634-016-0325-3","volume":"2016","author":"J Zhang","year":"2016","unstructured":"J. Zhang, X. Qiu, C. Shi, Y. Wu, Cognitive radar ambiguity function optimization for unimodular sequence. EURASIP J. Adv. Signal Process. 2016, 1\u201313 (2016)","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"1054_CR23","doi-asserted-by":"crossref","unstructured":"N.U.R. Junejo, M. Sattar, S. Adnan, H. Sun, A.B.M. Adam, A. Hassan, H. Esmaiel, A survey on physical layer techniques and challenges in underwater communication systems. J. Marine Sci. Eng. 11(4) (2023)","DOI":"10.3390\/jmse11040885"},{"key":"1054_CR24","doi-asserted-by":"crossref","unstructured":"X. Li, C. Yang, J. Song, S. Feng, W. Li, H. He, A motion control method for agent based on dyna-q algorithm, in 2023 4th International Conference on Computer Engineering and Application (ICCEA), pp. 274\u2013278 (2023)","DOI":"10.1109\/ICCEA58433.2023.10135395"}],"container-title":["EURASIP Journal on Advances in Signal Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-023-01054-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13634-023-01054-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-023-01054-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T23:51:31Z","timestamp":1730505091000},"score":1,"resource":{"primary":{"URL":"https:\/\/asp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13634-023-01054-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,14]]},"references-count":24,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["1054"],"URL":"https:\/\/doi.org\/10.1186\/s13634-023-01054-7","relation":{},"ISSN":["1687-6180"],"issn-type":[{"value":"1687-6180","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,14]]},"assertion":[{"value":"27 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 September 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All procedures performed in this paper were in accordance with the ethical standards of research. community","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"116"}}