{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,3]],"date-time":"2026-10-03T16:55:00Z","timestamp":1791046500236,"version":"4.1.0"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,3,29]],"date-time":"2024-03-29T00:00:00Z","timestamp":1711670400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>A strategy refers to the rules that the agent chooses the available actions to achieve goals. Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system\u2019s utility, decrease the overall cost, and increase mission success probability. This article proposes a novel hierarchical strategy decomposition approach based on Bayesian chaining to separate an intricate policy into several simple sub-policies and organize their relationships as Bayesian strategy networks (BSN). We integrate this approach into the state-of-the-art DRL method\u2014soft actor-critic (SAC), and build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy. Our method achieves the state-of-the-art performance on the standard continuous control benchmarks in the OpenAI Gym environment. The results demonstrate that the promising potential of the BSAC method significantly improves training efficiency. Furthermore, we extend the topic to the Multi-Agent systems (MAS), discussing the potential research fields and directions.<\/jats:p>","DOI":"10.1145\/3643862","type":"journal-article","created":{"date-parts":[[2024,2,1]],"date-time":"2024-02-01T06:56:39Z","timestamp":1706770599000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Bayesian Strategy Networks Based Soft Actor-Critic Learning"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5342-1798","authenticated-orcid":false,"given":"Qin","family":"Yang","sequence":"first","affiliation":[{"name":"Bradley University, Peoria, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5451-8209","authenticated-orcid":false,"given":"Ramviyas","family":"Parasuraman","sequence":"additional","affiliation":[{"name":"University of Georgia, Athens, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,29]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.01.002"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2743240"},{"key":"e_1_3_3_4_2","article-title":"Gradient descent for general reinforcement learning","author":"Baird Leemon","year":"1998","unstructured":"Leemon Baird and Andrew Moore. 1998. Gradient descent for general reinforcement learning. InProceedings of the 11th International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 11th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_3_5_2","unstructured":"Gabriel Barth-Maron Matthew W. Hoffman David Budden Will Dabney Dan Horgan Dhruva TB Alistair Muldal Nicolas Heess and Timothy Lillicrap. 2018. Distributional policy gradients. In International Conference on Learning Representations."},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecolecon.2008.02.012"},{"key":"e_1_3_3_7_2","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:1606.01540. Retrieved from https:\/\/arxiv.org\/abs\/1606.01540"},{"key":"e_1_3_3_8_2","first-page":"1534","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence","volume":"2003","author":"Buro Michael","year":"2003","unstructured":"Michael Buro. 2003. Real-time strategy games: A new AI research challenge. In Proceedings of the International Joint Conference on Artificial Intelligence. Vol. 2003, 1534\u20131535."},{"key":"e_1_3_3_9_2","volume-title":"Strategy: A History","author":"Freedman Lawrence","year":"2015","unstructured":"Lawrence Freedman. 2015. Strategy: A History. Oxford University Press."},{"key":"e_1_3_3_10_2","first-page":"1587","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Fujimoto Scott","year":"2018","unstructured":"Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning. PMLR, 1587\u20131596."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1561\/2200000049"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2012.2218595"},{"key":"e_1_3_3_13_2","unstructured":"Shixiang Gu Timothy Lillicrap Zoubin Ghahramani Richard E. Turner and Sergey Levine. 2017. Q-Prop: Sample-efficient policy gradient with An off-policy critic. In International Conference on Learning Representations."},{"key":"e_1_3_3_14_2","first-page":"1352","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Haarnoja Tuomas","year":"2017","unstructured":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. 2017. Reinforcement learning with deep energy-based policies. In Proceedings of the International Conference on Machine Learning. PMLR, 1352\u20131361."},{"key":"e_1_3_3_15_2","first-page":"1861","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Haarnoja Tuomas","year":"2018","unstructured":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning. PMLR, 1861\u20131870."},{"key":"e_1_3_3_16_2","unstructured":"Tuomas Haarnoja Aurick Zhou Kristian Hartikainen George Tucker Sehoon Ha Jie Tan Vikash Kumar Henry Zhu Abhishek Gupta Pieter Abbeel and Sergey Levine. 2018. Soft actor-critic algorithms and applications. arXiv:1812.05905. Retrieved from https:\/\/arxiv.org\/abs\/1812.05905"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-27645-3_9"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-019-09421-1"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2022.3155327"},{"key":"e_1_3_3_21_2","unstructured":"Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.5555\/1795555"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogsys.2006.07.004"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2012.12.051"},{"key":"e_1_3_3_25_2","unstructured":"Yuxi Li. 2017. Deep reinforcement learning: An overview. arXiv:1701.07274. Retrieved from https:\/\/arxiv.org\/abs\/1701.07274"},{"key":"e_1_3_3_26_2","unstructured":"Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. In International Conference on Learning Representations."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.3390\/robotics10010022"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","unstructured":"Siqi Liu Guy Lever Zhe Wang Josh Merel S. M. Ali Eslami Daniel Hennes Wojciech M. Czarnecki Yuval Tassa Shayegan Omidshafiei Abbas Abdolmaleki Noah Y. Siegel Leonard Hasenclever Luke Marris Saran Tunyasuvunakool H. Francis Song Markus Wulfmeier Paul Muller Tuomas Haarnoja Brendan D. Tracey Karl Tuyls Thore Graepel and Nicolas Heess. 2021. From motor control to team play in simulated humanoid football. arXiv:2105.12196. Retrieved from https:\/\/arxiv.org\/abs\/2105.12196","DOI":"10.1126\/scirobotics.abo0235"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecolmodel.2017.05.011"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.envsoft.2018.09.016"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888912000057"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2010-343"},{"key":"e_1_3_3_33_2","first-page":"1928","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Mnih Volodymyr","year":"2016","unstructured":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1928\u20131937."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9407.001.0001"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/MPRV.2005.13"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453160"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-017-0468-y"},{"key":"e_1_3_3_39_2","first-page":"1","volume-title":"Proceedings of the International Symposium on Artificial Intelligence and Mathematics,","author":"Poupart Pascal","year":"2008","unstructured":"Pascal Poupart and Nikos Vlassis. 2008. Model-based Bayesian reinforcement learning in partially observable domains. In Proceedings of the International Symposium on Artificial Intelligence and Mathematics,. 1\u20132."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143932"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2018.2840971"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13748-019-00194-y"},{"key":"e_1_3_3_43_2","first-page":"1889","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Schulman John","year":"2015","unstructured":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In Proceedings of the International Conference on Machine Learning. PMLR, 1889\u20131897."},{"key":"e_1_3_3_44_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. Retrieved from https:\/\/arxiv.org\/abs\/1707.06347"},{"key":"e_1_3_3_45_2","unstructured":"Kun Shao Zhentao Tang Yuanheng Zhu Nannan Li and Dongbin Zhao. 2019. A survey of deep reinforcement learning in video games. arXiv:1912.10944. Retrieved from https:\/\/arxiv.org\/abs\/1912.10944"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-021-09997-9"},{"key":"e_1_3_3_47_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press."},{"key":"e_1_3_3_48_2","article-title":"Learning diverse skills via maximum entropy deep reinforcement learning","author":"Tang Haoran","year":"2017","unstructured":"Haoran Tang and Tuomas Haarnoja. 2017. Learning diverse skills via maximum entropy deep reinforcement learning. Berkeley Artificial Intelligence Research. Retrieved Oct 6, 2017 from https:\/\/bair.berkeley.edu\/blog\/2017\/10\/06\/soft-q-learning\/","journal-title":"Berkeley Artificial Intelligence Research"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11798"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2020.100022"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.106657"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-16-8044-1_14"},{"key":"e_1_3_3_54_2","first-page":"1995","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Wang Ziyu","year":"2016","unstructured":"Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling network architectures for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1995\u20132003."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.50.221"},{"key":"e_1_3_3_56_2","first-page":"5040","article-title":"Self-adaptive swarm system (SASS)","author":"Yang Qin","year":"2021","unstructured":"Qin Yang. 2021. Self-adaptive swarm system (SASS). In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI-21, 5040\u20135041. Doctoral Consortium.","journal-title":"Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI-21"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.5555\/AAI28968673"},{"key":"e_1_3_3_58_2","article-title":"Hierarchical needs-driven agent learning systems: From deep reinforcement learning to diverse strategies","author":"Yang Qin","year":"2023","unstructured":"Qin Yang. 2023. Hierarchical needs-driven agent learning systems: From deep reinforcement learning to diverse strategies. InProceedings of the 37th AAAI 2023 Conference on Artificial Intelligence and Robotics Bridge Program.","journal-title":"Proceedings of the 37th AAAI 2023 Conference on Artificial Intelligence and Robotics Bridge Program"},{"key":"e_1_3_3_59_2","doi-asserted-by":"crossref","unstructured":"Qin Yang and Rui Liu. 2023. Understanding the application of utility theory in robotics and artificial intelligence: A survey. arXiv:2306.09445. Retrieved from https:\/\/arxiv.org\/abs\/2306.09445","DOI":"10.2139\/ssrn.4538600"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/MRS.2019.8901075"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC42975.2020.9283249"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/SSRR50563.2020.9292570"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC52423.2021.9659187"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3555776.3577642"},{"key":"e_1_3_3_65_2","doi-asserted-by":"crossref","unstructured":"Qin Yang and Ramviyas Parasuraman. 2023. A strategy-oriented bayesian soft actor-critic model. Procedia Computer Science 220 (2023) 561\u2013566.","DOI":"10.1016\/j.procs.2023.03.071"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/7588221"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI47803.2020.9308468"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643862","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643862","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:57:34Z","timestamp":1750276654000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643862"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,29]]},"references-count":66,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3643862"],"URL":"https:\/\/doi.org\/10.1145\/3643862","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,29]]},"assertion":[{"value":"2023-08-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}