{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T17:05:59Z","timestamp":1768583159042,"version":"3.49.0"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"name":"NSF CISE","award":["2302720, 2312758, 2038029"],"award-info":[{"award-number":["2302720, 2312758, 2038029"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2026,2,28]]},"abstract":"<jats:p>\n                    Developing grounding techniques for LLMs poses two requirements for interactive environments, i.e., (i) the presence of rich knowledge beyond the scope of existing LLMs and (ii) the complexity of tasks that require strategic reasoning. Existing environments fail to meet both requirements due to their simplicity or reliance on commonsense knowledge already encoded in LLMs for interaction. In this article, we present Pok\u00e9LLMon, a new benchmark enriched with fictional game knowledge and characterized by the intense, dynamic, and adversarial gameplay of Pok\u00e9mon battles, setting new challenges for the development of grounding and reasoning techniques in interactive environments. Empirical evaluations demonstrate that existing LLMs lack game knowledge and struggle in Pok\u00e9mon battles. We investigate grounding techniques that leverage feedback and game knowledge, and provide a thorough analysis of reasoning methods from a new perspective of action consistency. Additionally, we introduce higher-level reasoning challenges when playing against human players. The implementation of our benchmark is released at:\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/git-disl\/PokeLLMon\">https:\/\/github.com\/git-disl\/PokeLLMon<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1145\/3771095","type":"journal-article","created":{"date-parts":[[2025,10,7]],"date-time":"2025-10-07T11:19:49Z","timestamp":1759835989000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Pok\u00e9LLMon: A Grounding and Reasoning Benchmark for Large Language Models in Pok\u00e9mon Battles"],"prefix":"10.1145","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3297-6991","authenticated-orcid":false,"given":"Sihao","family":"Hu","sequence":"first","affiliation":[{"name":"School of Computer Science, Georgia Institute of Technology","place":["Atlanta, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4557-1865","authenticated-orcid":false,"given":"Tiansheng","family":"Huang","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology School of Computer Science","place":["Atlanta, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9048-9274","authenticated-orcid":false,"given":"Gaowen","family":"Liu","sequence":"additional","affiliation":[{"name":"Cisco Systems Inc","place":["San Jose, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7559-8997","authenticated-orcid":false,"given":"Ramana","family":"Kompella","sequence":"additional","affiliation":[{"name":"Cisco Systems Inc","place":["San Jose, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4138-3082","authenticated-orcid":false,"given":"Ling","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science, Georgia Institute of Technology","place":["Atlanta, United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,1,14]]},"reference":[{"key":"e_1_3_4_2_2","unstructured":"Bulbapedia: The community-driven Pok\u00e9mon encyclopedia. Retrieved Feb 1 2025 from https:\/\/bulbapedia.bulbagarden.net\/wiki\/Main_Page"},{"key":"e_1_3_4_3_2","unstructured":"Pokemon Wiki. Retrieved Feb 1 2025 from https:\/\/pokemon.fandom.com\/wiki\/Pok\u00e9mon_Wiki"},{"key":"e_1_3_4_4_2","unstructured":"2024. List of Moves. (2024). Retrieved Feb 1 2025 from https:\/\/bulbapedia.bulbagarden.net\/wiki\/List_of_moves"},{"key":"e_1_3_4_5_2","unstructured":"2024. List of Pok\u00e9mon by National Pok\u00e9dex Number. (2024). Retrieved Feb 1 2025 from https:\/\/bulbapedia.bulbagarden.net\/wiki\/List_of_Pok~Al'mon_by_National_Pok~Al'dex_number"},{"key":"e_1_3_4_6_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et\u00a0al. 2023. Gpt-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_4_7_2","unstructured":"Saaket Agashe Yue Fan and Xin Eric Wang. 2023. Evaluating multi-agent coordination abilities in large language models. arXiv:2310.03903. Retrieved Feb 1 2025 from https:\/\/arxiv.org\/abs\/2310.03903"},{"key":"e_1_3_4_8_2","unstructured":"Christopher Berner Greg Brockman Brooke Chan Vicki Cheung Przemys\u0142aw D\u0119biak Christy Dennison David Farhi Quirin Fischer Shariq Hashme Chris Hesse et\u00a0al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv:1912.06680. Retrieved from https:\/\/arxiv.org\/abs\/1912.06680"},{"key":"e_1_3_4_9_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_10_2","doi-asserted-by":"crossref","unstructured":"Laura Cabello Jiaang Li and Ilias Chalkidis. 2023. PokemonChat: Auditing ChatGPT for Pok \\(\\backslash\\) \u2019emon universe knowledge. arXiv:2306.03024. Retrieved from https:\/\/arxiv.org\/abs\/2306.03024","DOI":"10.2139\/ssrn.4396798"},{"key":"e_1_3_4_11_2","article-title":"On the utility of learning about humans for human-ai coordination","volume":"32","author":"Carroll Micah","year":"2019","unstructured":"Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_12_2","unstructured":"Thomas Carta Cl\u00e9ment Romac Thomas Wolf Sylvain Lamprier Olivier Sigaud and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. arXiv:2302.02662. Retrieved from https:\/\/arxiv.org\/abs\/2302.02662"},{"key":"e_1_3_4_13_2","unstructured":"Yicheng Feng Yuxuan Wang Jiazheng Liu Sipeng Zheng and Zongqing Lu. 2023. Llama rider: Spurring large language models to explore the open world. arXiv:2310.08922. Retrieved from https:\/\/arxiv.org\/abs\/2310.08922"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2005.08.041"},{"key":"e_1_3_4_15_2","unstructured":"Jiaxian Guo Bo Yang Paul Yoo Bill Yuchen Lin Yusuke Iwasawa and Yutaka Matsuo. 2023. Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4. arXiv:2309.17277. Retrieved from https:\/\/arxiv.org\/abs\/2309.17277"},{"key":"e_1_3_4_16_2","unstructured":"Danijar Hafner. 2021. Benchmarking the spectrum of agent capabilities. arXiv:2109.06780. Retrieved from https:\/\/arxiv.org\/abs\/2109.06780"},{"key":"e_1_3_4_17_2","unstructured":"Sihao Hu Tiansheng Huang Fatih Ilhan Selim Tekin Gaowen Liu Ramana Kompella and Ling Liu. 2024. A survey on large language model-based game agents. arXiv:2404.02039. Retrieved from https:\/\/arxiv.org\/abs\/2404.02039"},{"key":"e_1_3_4_18_2","unstructured":"Weiyu Ma Qirui Mi Xue Yan Yuqiao Wu Runji Lin Haifeng Zhang and Jun Wang. 2023. Large language models play starcraft II: Benchmarks and a chain of summarization approach. arXiv:2312.11865. Retrieved from https:\/\/arxiv.org\/abs\/2312.11865"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2024.01.011"},{"key":"e_1_3_4_20_2","unstructured":"Chevalier-Boisvert Maxime Dzmitry Bahdanau Salem Lahlou Lucas Willems Chitwan Saharia Thien Huu Nguyen and Yoshua Bengio. 2018. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv:1810.08272. Retrieved from https:\/\/arxiv.org\/abs\/1810.08272"},{"key":"e_1_3_4_21_2","article-title":"Minecraft","author":"Studios Mojang","unstructured":"Mojang Studios. Minecraft. Retrieved Feb, 1, 2025 from https:\/\/www.minecraft.net\/en-us","journal-title":"https:\/\/www.minecraft.net\/en-us"},{"key":"e_1_3_4_22_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et\u00a0al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730\u201327744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00886"},{"key":"e_1_3_4_24_2","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. Retrieved from https:\/\/arxiv.org\/abs\/1707.06347"},{"key":"e_1_3_4_25_2","unstructured":"Noah Shinn Beck Labash and Ashwin Gopinath. 2023. Reflexion: An autonomous agent with dynamic memory and self-reflection. arXiv:2303.11366. Retrieved from https:\/\/arxiv.org\/abs\/2303.11366"},{"key":"e_1_3_4_26_2","unstructured":"Mohit Shridhar Xingdi Yuan Marc-Alexandre C\u00f4t\u00e9 Yonatan Bisk Adam Trischler and Matthew Hausknecht. 2020. Alfworld: Aligning text and embodied environments for interactive learning. arXiv:2010.03768. Retrieved from https:\/\/arxiv.org\/abs\/2010.03768"},{"key":"e_1_3_4_27_2","unstructured":"Weihao Tan Wentao Zhang Shanqi Liu Longtao Zheng Xinrun Wang and Bo An. 2024. True knowledge comes from practice: Aligning LLMs with embodied environments via reinforcement learning. arXiv:2401.14151. Retrieved from https:\/\/arxiv.org\/abs\/2401.14151"},{"key":"e_1_3_4_28_2","unstructured":"Oriol Vinyals Timo Ewalds Sergey Bartunov Petko Georgiev Alexander Sasha Vezhnevets Michelle Yeo Alireza Makhzani Heinrich K\u00fcttler John Agapiou Julian Schrittwieser et\u00a0al. 2017. Starcraft ii: A new challenge for reinforcement learning. arXiv:1708.04782. Retrieved from https:\/\/arxiv.org\/abs\/1708.04782"},{"key":"e_1_3_4_29_2","unstructured":"Guanzhi Wang Yuqi Xie Yunfan Jiang Ajay Mandlekar Chaowei Xiao Yuke Zhu Linxi Fan and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv:2305.16291. Retrieved from https:\/\/arxiv.org\/abs\/2305.16291"},{"key":"e_1_3_4_30_2","unstructured":"Lei Wang Chen Ma Xueyang Feng Zeyu Zhang Hao Yang Jingsen Zhang Zhiyuan Chen Jiakai Tang Xu Chen Yankai Lin et\u00a0al. 2023. A survey on large language model based autonomous agents. arXiv:2308.11432. Retrieved from https:\/\/arxiv.org\/abs\/2308.11432"},{"key":"e_1_3_4_31_2","doi-asserted-by":"crossref","unstructured":"Ruoyao Wang Peter Jansen Marc-Alexandre C\u00f4t\u00e9 and Prithviraj Ammanabrolu. 2022. ScienceWorld: Is your agent smarter than a 5th grader? arXiv:2203.07540. Retrieved from https:\/\/arxiv.org\/abs\/2203.07540","DOI":"10.18653\/v1\/2022.emnlp-main.775"},{"key":"e_1_3_4_32_2","unstructured":"Xuezhi Wang Jason Wei Dale Schuurmans Quoc Le Ed Chi Sharan Narang Aakanksha Chowdhery and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Retrieved from https:\/\/arxiv.org\/abs\/2203.11171"},{"key":"e_1_3_4_33_2","unstructured":"Zihao Wang Shaofei Cai Anji Liu Yonggang Jin Jinbing Hou Bowei Zhang Haowei Lin Zhaofeng He Zilong Zheng Yaodong Yang et\u00a0al. 2023. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv:2311.05997. Retrieved from https:\/\/arxiv.org\/abs\/2311.05997"},{"key":"e_1_3_4_34_2","unstructured":"Zihao Wang Shaofei Cai Anji Liu Xiaojian Ma and Yitao Liang. 2023. Describe explain plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv:2302.01560. Retrieved from https:\/\/arxiv.org\/abs\/2302.01560"},{"key":"e_1_3_4_35_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et\u00a0al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_36_2","unstructured":"Wikipedia. 2023. Gameplay of Pok\u00e9mon. (2023). Retrieved Feb 1 2025 from https:\/\/en.wikipedia.org\/wiki\/Gameplay_of_Pok%C3%A9mon"},{"key":"e_1_3_4_37_2","unstructured":"Zhiheng Xi Wenxiang Chen Xin Guo Wei He Yiwen Ding Boyang Hong Ming Zhang Junzhe Wang Senjie Jin Enyu Zhou et\u00a0al. 2023. The rise and potential of large language model based agents: A survey. arXiv:2309.07864. Retrieved from https:\/\/arxiv.org\/abs\/2309.07864"},{"key":"e_1_3_4_38_2","article-title":"Language models meet world models: Embodied experiences enhance language models","volume":"36","author":"Xiang Jiannan","year":"2024","unstructured":"Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. 2024. Language models meet world models: Embodied experiences enhance language models. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_39_2","unstructured":"Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas L. Griffiths Yuan Cao and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv:2305.10601. Retrieved from https:\/\/arxiv.org\/abs\/2305.10601"},{"key":"e_1_3_4_40_2","unstructured":"Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv:2210.03629. Retrieved from https:\/\/arxiv.org\/abs\/2210.03629"},{"key":"e_1_3_4_41_2","unstructured":"Ceyao Zhang Kaijie Yang Siyi Hu Zihao Wang Guanghe Li Yihang Sun Cheng Zhang Zhaowei Zhang Anji Liu Song-Chun Zhu et\u00a0al. 2023. Proagent: Building proactive cooperative ai with large language models. arXiv:2308.11339. Retrieved from https:\/\/arxiv.org\/abs\/2308.11339"},{"key":"e_1_3_4_42_2","unstructured":"Xizhou Zhu Yuntao Chen Hao Tian Chenxin Tao Weijie Su Chenyu Yang Gao Huang Bin Li Lewei Lu Xiaogang Wang et\u00a0al. 2023. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv:2305.17144. Retrieved from https:\/\/arxiv.org\/abs\/2305.17144"}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3771095","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T05:08:12Z","timestamp":1768540092000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3771095"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,14]]},"references-count":41,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,2,28]]}},"alternative-id":["10.1145\/3771095"],"URL":"https:\/\/doi.org\/10.1145\/3771095","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"value":"1533-5399","type":"print"},{"value":"1557-6051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,14]]},"assertion":[{"value":"2025-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-25","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}