{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T07:22:45Z","timestamp":1760080965366,"version":"3.41.2"},"reference-count":30,"publisher":"World Scientific Pub Co Pte Ltd","issue":"02","funder":[{"DOI":"10.13039\/501100001381","name":"National Research Foundation Singapore","doi-asserted-by":"publisher","award":["ISG2-GC-2023-009"],"award-info":[{"award-number":["ISG2-GC-2023-009"]}],"id":[{"id":"10.13039\/501100001381","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Artif. Intell. Robot. Res."],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p> Despite the impressive performance across numerous tasks, Large Language Models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, Reinforcement Learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. First, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with LOw-Rank Adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. (i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. (ii) Benefiting from LLMs\u2019 open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. (iii) Under our framework, there is no significant loss of the LLMs\u2019 original ability during online PPO finetuning.<jats:sup>a<\/jats:sup> <\/jats:p>","DOI":"10.1142\/s2972335324500042","type":"journal-article","created":{"date-parts":[[2024,5,31]],"date-time":"2024-05-31T07:37:38Z","timestamp":1717141058000},"source":"Crossref","is-referenced-by-count":3,"title":["TWOSOME: An Efficient Online Framework to Align LLMs with Embodied Environments via Reinforcement Learning"],"prefix":"10.1142","volume":"01","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-4231-8777","authenticated-orcid":false,"given":"Weihao","family":"Tan","sequence":"first","affiliation":[{"name":"College of Computing and Data Science, Nanyang Technological University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2767-6998","authenticated-orcid":false,"given":"Wentao","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Computing and Data Science, Nanyang Technological University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0583-2423","authenticated-orcid":false,"given":"Shanqi","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Zhejiang University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7301-0862","authenticated-orcid":false,"given":"Longtao","family":"Zheng","sequence":"additional","affiliation":[{"name":"College of Computing and Data Science, Nanyang Technological University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3369-219X","authenticated-orcid":false,"given":"Xinrun","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Computing and Data Science, Nanyang Technological University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7064-7438","authenticated-orcid":false,"given":"Bo","family":"An","sequence":"additional","affiliation":[{"name":"College of Computing and Data Science, Nanyang Technological University, Singapore"},{"name":"Skywork AI, 2 Science Park Drive, Ascent Bridge+ #01-08, Singapore 118222, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2024,7,29]]},"reference":[{"key":"S2972335324500042BIB001","first-page":"1877","volume-title":"Proc. 34th Int. Conf. Neural Information Processing Systems","author":"Brown T. B.","year":"2020"},{"volume-title":"11th Int. Conf. Learning Representations","year":"2023","author":"Yao S.","key":"S2972335324500042BIB005"},{"key":"S2972335324500042BIB008","first-page":"287","volume-title":"Conf. Robot Learning","author":"Brohan A.","year":"2023"},{"volume-title":"Reinforcement Learning: An Introduction","year":"2018","author":"Sutton R. S.","key":"S2972335324500042BIB010"},{"volume-title":"11th Int. Conf. Learning Representations","year":"2023","author":"Kumar A.","key":"S2972335324500042BIB011"},{"volume-title":"Int. Conf. Learning Representations","year":"2022","author":"Hu E. J.","key":"S2972335324500042BIB014"},{"key":"S2972335324500042BIB016","first-page":"1379","volume-title":"Conf. Robot Learning","author":"Karamcheti S.","year":"2021"},{"key":"S2972335324500042BIB017","first-page":"31199","volume-title":"Advances in Neural Information Processing Systems","author":"Li S.","year":"2022"},{"key":"S2972335324500042BIB018","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00169"},{"key":"S2972335324500042BIB019","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58539-6_16"},{"key":"S2972335324500042BIB020","first-page":"17359","volume-title":"Int. Conf. Machine Learning","author":"Parisi S.","year":"2022"},{"key":"S2972335324500042BIB022","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01441"},{"key":"S2972335324500042BIB023","first-page":"1531","volume-title":"Conf. Robot Learning","author":"Ren A. Z.","year":"2023"},{"key":"S2972335324500042BIB025","first-page":"894","volume-title":"Conf. Robot Learning","author":"Shridhar M.","year":"2022"},{"key":"S2972335324500042BIB026","first-page":"9118","volume-title":"Int. Conf. Machine Learning","author":"Huang W.","year":"2022"},{"key":"S2972335324500042BIB027","first-page":"1769","volume-title":"Proceedings of The 6th Conference on Robot Learning","volume":"205","author":"Huang W."},{"volume-title":"2nd Workshop on Language and Reinforcement Learning","year":"2022","author":"Dasgupta I.","key":"S2972335324500042BIB034"},{"key":"S2972335324500042BIB038","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-023-00626-4"},{"key":"S2972335324500042BIB039","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"S2972335324500042BIB040","first-page":"4582","volume-title":"Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int. Joint Conf. Natural Language Processing","volume":"1","author":"Lisa Li X.","year":"2021"},{"key":"S2972335324500042BIB041","first-page":"2790","volume-title":"Int. Conf. Machine Learning","author":"Houlsby N.","year":"2019"},{"key":"S2972335324500042BIB042","first-page":"27730","volume-title":"Advances in Neural Information Processing Systems","author":"Ouyang L.","year":"2022"},{"key":"S2972335324500042BIB044","volume-title":"Advances in Neural Information Processing Systems","volume":"16","author":"Theocharous G.","year":"2003"},{"key":"S2972335324500042BIB045","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v25i1.7982"},{"issue":"1","key":"S2972335324500042BIB046","first-page":"5485","volume":"21","author":"Raffel C.","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"S2972335324500042BIB047","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i14.17520"},{"issue":"1","key":"S2972335324500042BIB049","first-page":"12585","volume":"23","author":"Huang S.","year":"2022","journal-title":"J. Mach. Learn. Res."},{"key":"S2972335324500042BIB050","first-page":"4385","volume-title":"Advances in Neural Information Processing Systems","author":"Xiao Y.","year":"2022"},{"key":"S2972335324500042BIB051","doi-asserted-by":"publisher","DOI":"10.1111\/tops.12525"},{"key":"S2972335324500042BIB052","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00886"}],"container-title":["International Journal of Artificial Intelligence and Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S2972335324500042","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,9]],"date-time":"2024-09-09T03:23:21Z","timestamp":1725852201000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S2972335324500042"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6]]},"references-count":30,"journal-issue":{"issue":"02","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1142\/S2972335324500042"],"URL":"https:\/\/doi.org\/10.1142\/s2972335324500042","relation":{},"ISSN":["2972-3353","2972-3361"],"issn-type":[{"type":"print","value":"2972-3353"},{"type":"electronic","value":"2972-3361"}],"subject":[],"published":{"date-parts":[[2024,6]]},"article-number":"2450004"}}