{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T14:24:00Z","timestamp":1753885440206,"version":"3.41.2"},"reference-count":24,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2021,7,14]],"date-time":"2021-07-14T00:00:00Z","timestamp":1626220800000},"content-version":"vor","delay-in-days":194,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61902093"],"award-info":[{"award-number":["61902093"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003453","name":"Natural Science Foundation of Guangdong Province","doi-asserted-by":"publisher","award":["2020A1515010652"],"award-info":[{"award-number":["2020A1515010652"]}],"id":[{"id":"10.13039\/501100003453","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Complexity"],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:p>In recent years, deep reinforcement learning (DRL) achieves great success in many fields, especially in the field of games, such as AlphaGo, AlphaZero, and AlphaStar. However, due to the reward sparsity problem, the traditional DRL\u2010based method shows limited performance in 3D games, which contain much higher dimension of state space. To solve this problem, in this paper, we propose an intrinsic\u2010based policy optimization (IBPO) algorithm for reward sparsity. In the IBPO, a novel intrinsic reward is integrated into the value network, which provides an additional reward in the environment with sparse reward, so as to accelerate the training. Besides, to deal with the problem of value estimation bias, we further design three types of auxiliary tasks, which can evaluate the state value and the action more accurately in 3D scenes. Finally, a framework of auxiliary intrinsic\u2010based policy optimization (AIBPO) is proposed, which improves the performance of the IBPO. The experimental results show that the method is able to deal with the reward sparsity problem effectively. Therefore, the proposed method may be applied to real\u2010world scenarios, such as 3\u2010dimensional navigation and automatic driving, which can improve the sample utilization to reduce the cost of interactive sample collected by the real equipment.<\/jats:p>","DOI":"10.1155\/2021\/6698231","type":"journal-article","created":{"date-parts":[[2021,7,14]],"date-time":"2021-07-14T18:50:07Z","timestamp":1626288607000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["AIBPO: Combine the Intrinsic Reward and Auxiliary Task for 3D Strategy Game"],"prefix":"10.1155","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9168-5038","authenticated-orcid":false,"given":"Huale","family":"Li","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rui","family":"Cao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3512-0649","authenticated-orcid":false,"given":"Xuan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaohan","family":"Hou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tao","family":"Qian","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1986-9174","authenticated-orcid":false,"given":"Fengwei","family":"Jia","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6611-2046","authenticated-orcid":false,"given":"Jiajia","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6903-145X","authenticated-orcid":false,"given":"Shuhan","family":"Qi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2021,7,14]]},"reference":[{"key":"e_1_2_8_1_2","unstructured":"MnihV. KavukcuogluK. SilverD.et al. Playing atari with deep reinforce-ment learning 2013 https:\/\/arxiv.org\/abs\/1312.5602."},{"key":"e_1_2_8_2_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_2_8_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_2_8_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"e_1_2_8_5_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-03051-4"},{"key":"e_1_2_8_6_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.3912"},{"key":"e_1_2_8_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TG.2018.2834566"},{"key":"e_1_2_8_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TG.2018.2877047"},{"key":"e_1_2_8_9_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aau6249"},{"key":"e_1_2_8_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2018.2823329"},{"key":"e_1_2_8_11_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z"},{"key":"e_1_2_8_12_2","doi-asserted-by":"publisher","DOI":"10.1002\/cplx.20036"},{"key":"e_1_2_8_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/tip.2016.2563981"},{"key":"e_1_2_8_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/tnnls.2018.2805379"},{"key":"e_1_2_8_15_2","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/8815770"},{"key":"e_1_2_8_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2934906"},{"key":"e_1_2_8_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.08.024"},{"key":"e_1_2_8_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05845-8"},{"key":"e_1_2_8_19_2","unstructured":"MnihV. BadiaA. P. MirzaM.et al. Asynchronous methods for deep reinforcement learning Proceedings of the International Conference on Machine Learning June 2016 New York NY USA 1928\u20131937."},{"key":"e_1_2_8_20_2","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015015"},{"key":"e_1_2_8_21_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_2_8_22_2","doi-asserted-by":"crossref","unstructured":"LampleG.andChaplotD. S. Playing fps games with deep reinforcement learning Proceedings of the 31st AAAI Conference on Artificial Intelligence 2017 San Francisco CA USA 2140\u20132146.","DOI":"10.1609\/aaai.v31i1.10827"},{"key":"e_1_2_8_23_2","unstructured":"DosovitskiyA.andKoltunV. Learning to act by predicting the future Proceedings of the International Conference on Learning Representations 2017 Toulon France 637\u2013645."},{"key":"e_1_2_8_24_2","doi-asserted-by":"crossref","unstructured":"HesselM. ModayilJ. Van HasseltH.et al. Rainbow: combining improvements in deep reinforcement learning Proceedings of the AAAI Conference on Artificial Intelligence 2018 New Orleans LA USA 1147\u20131153.","DOI":"10.1609\/aaai.v32i1.11796"}],"container-title":["Complexity"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/complexity\/2021\/6698231.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/complexity\/2021\/6698231.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1155\/2021\/6698231","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T21:52:05Z","timestamp":1723240325000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1155\/2021\/6698231"}},"subtitle":[],"editor":[{"given":"Abd E.I.-Baset","family":"Hassanien","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,1]]},"references-count":24,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["10.1155\/2021\/6698231"],"URL":"https:\/\/doi.org\/10.1155\/2021\/6698231","archive":["Portico"],"relation":{},"ISSN":["1076-2787","1099-0526"],"issn-type":[{"type":"print","value":"1076-2787"},{"type":"electronic","value":"1099-0526"}],"subject":[],"published":{"date-parts":[[2021,1]]},"assertion":[{"value":"2020-10-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"6698231"}}