{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T10:10:56Z","timestamp":1775470256067,"version":"3.50.1"},"reference-count":50,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,8,26]],"date-time":"2022-08-26T00:00:00Z","timestamp":1661472000000},"content-version":"vor","delay-in-days":237,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,8,26]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Reinforcement Learning has shown success in a number of complex virtual environments. However, many challenges still exist towards solving problems with natural language as a core component. Interactive Fiction Games (or Text Games) are one such problem type that offer a set of safe, partially observable environments where natural language is required as part of the Reinforcement Learning solution. Therefore, this survey\u2019s aim is to assist in the development of new Text Game problem settings and solutions for Reinforcement Learning informed by natural language. Specifically, this survey: 1) introduces the challenges in Text Game Reinforcement Learning problems, 2) outlines the generation tools for rendering Text Games and the subsequent environments generated, and 3) compares the agent architectures currently applied to provide a systematic review of benchmark methodologies and opportunities for future researchers.<\/jats:p>","DOI":"10.1162\/tacl_a_00495","type":"journal-article","created":{"date-parts":[[2022,8,26]],"date-time":"2022-08-26T17:25:38Z","timestamp":1661534738000},"page":"873-887","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":12,"title":["A Survey of Text Games for Reinforcement Learning Informed by Natural Language"],"prefix":"10.1162","volume":"10","author":[{"given":"Philip","family":"Osborne","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Manchester, United Kingdom. philiposbornedata@gmail.com"}]},{"given":"Heido","family":"N\u00f5mm","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Manchester, United Kingdom. heidonomm@gmail.com"}]},{"given":"Andr\u00e9","family":"Freitas","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Manchester, United Kingdom. andre.freitas@manchester.ac.uk"}]}],"member":"281","published-online":{"date-parts":[[2022,8,26]]},"reference":[{"key":"2022082617253123800_bib1","article-title":"Learning dynamic belief graphs to generalize on text-based games","volume":"33","author":"Adhikari","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022082617253123800_bib2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6228","article-title":"Ledeepchef: Deep reinforcement learning agent for families of text-based games","volume-title":"AAAI","author":"Adolphs","year":"2020"},{"key":"2022082617253123800_bib3","article-title":"Graph constrained reinforcement learning for natural language action spaces","author":"Ammanabrolu","year":"2020","journal-title":"International Conference on Learning Representations"},{"key":"2022082617253123800_bib4","doi-asserted-by":"publisher","first-page":"3557","DOI":"10.18653\/v1\/N19-1358","article-title":"Playing text-adventure games with graph-based deep reinforcement learning","author":"Ammanabrolu","year":"2019","journal-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"},{"key":"2022082617253123800_bib5","article-title":"How to avoid being eaten by a grue: Structured exploration strategies for textual worlds","author":"Ammanabrolu","year":"2020"},{"key":"2022082617253123800_bib6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.64","article-title":"How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds","author":"Ammanabrolu","year":"2020","journal-title":"ArXiv"},{"key":"2022082617253123800_bib7","article-title":"Zork: The great underground empire \u2013 part i","author":"Anderson","year":"1980"},{"key":"2022082617253123800_bib8","doi-asserted-by":"publisher","first-page":"344","DOI":"10.3115\/v1\/P15-1034","article-title":"Leveraging linguistic structure for open domain information extraction","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Angeli","year":"2015"},{"key":"2022082617253123800_bib9","doi-asserted-by":"publisher","first-page":"3002","DOI":"10.18653\/v1\/2020.emnlp-main.241","article-title":"Bootstrapped q-learning with context relevant observation pruning to generalize in text-based games","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020","author":"Chaudhury","year":"2020"},{"key":"2022082617253123800_bib10","article-title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","author":"Cho","year":"2014","journal-title":"Empirical Methods in Natural Language Processing"},{"key":"2022082617253123800_bib11","doi-asserted-by":"publisher","first-page":"520","DOI":"10.1145\/3366424.3384374","article-title":"Games for fairness and interpretability","volume-title":"Companion Proceedings of the Web Conference 2020","author":"Chu","year":"2020"},{"key":"2022082617253123800_bib12","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1007\/978-3-030-24337-1_3","article-title":"Textworld: A learning environment for text-based games","author":"C\u00f4t\u00e9","year":"2018","journal-title":"Workshop on Computer Games"},{"key":"2022082617253123800_bib13","article-title":"Building dynamic knowledge graphs from text using machine reading comprehension","author":"Das","year":"2019","journal-title":"ICLR"},{"key":"2022082617253123800_bib14","article-title":"Challenges of real-world reinforcement learning","author":"Dulac-Arnold","year":"2019","journal-title":"Proceedings of the 36th International Conference on Machine Learning"},{"key":"2022082617253123800_bib15","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/144","article-title":"What can you do with a rock? Affordance extraction via word embeddings","author":"Fulda","year":"2017","journal-title":"arXiv preprint arXiv: 1703.03429"},{"key":"2022082617253123800_bib16","article-title":"Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning","author":"Guo","year":"2020","journal-title":"EMNLP"},{"key":"2022082617253123800_bib17","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10295","article-title":"Deep reinforcement learning with double q-learning","author":"Hasselt","year":"2016","journal-title":"AAAI"},{"key":"2022082617253123800_bib18","article-title":"Interactive fiction games: A colossal adventure","author":"Hausknecht","year":"2019","journal-title":"ArXiv"},{"key":"2022082617253123800_bib19","article-title":"Nail: A general interactive fiction agent","author":"Hausknecht","year":"2019","journal-title":"arXiv preprint arXiv:1902.04259"},{"key":"2022082617253123800_bib20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1153","article-title":"Deep reinforcement learning with a natural language action space","author":"Ji","year":"2016","journal-title":"Association for Computational Linguistics (ACL)"},{"key":"2022082617253123800_bib21","article-title":"What would jiminy cricket do? Towards agents that behave morally","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Hendrycks","year":"2021"},{"issue":"8","key":"2022082617253123800_bib22","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2022082617253123800_bib23","doi-asserted-by":"publisher","first-page":"4328","DOI":"10.1609\/aaai.v34i04.5857","article-title":"Algorithmic improvements for deep reinforcement learning applied to interactive fiction.","volume-title":"AAAI","author":"Jain","year":"2020"},{"key":"2022082617253123800_bib24","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1109\/CIG.2017.8080433","article-title":"Text-based adventures of the Golovin AI agent","author":"Kostka","year":"2017","journal-title":"Computational Intelligence and Games (CIG)"},{"key":"2022082617253123800_bib25","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/880","article-title":"A survey of reinforcement learning informed by natural language","author":"Luketina","year":"2019","journal-title":"arXiv e-prints"},{"key":"2022082617253123800_bib26","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/207","article-title":"Exploration based language learning for text-based games","author":"Madotto","year":"2020","journal-title":"IJCAI"},{"key":"2022082617253123800_bib27","first-page":"1928","article-title":"Asynchronous methods for deep reinforcement learning","author":"Mnih","year":"2016","journal-title":"Proceedings of The 33rd International Conference on Machine Learning, PMLR 2016"},{"key":"2022082617253123800_bib28","article-title":"Playing Atari with deep reinforcement learning","author":"Mnih","year":"2013","journal-title":"CoRR"},{"key":"2022082617253123800_bib29","article-title":"Text-based RL agents with commonsense knowledge: New challenges, environments and baselines","author":"Murugesan","year":"2020","journal-title":"ArXiv"},{"key":"2022082617253123800_bib30","article-title":"Enhancing text-based reinforcement learning agents with commonsense knowledge","author":"Murugesan","year":"2020","journal-title":"arXiv preprint arXiv:2005.00811"},{"key":"2022082617253123800_bib31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1001","article-title":"Language understanding for text-based games using deep reinforcement learning","author":"Narasimhan","year":"2015","journal-title":"EMNLP"},{"key":"2022082617253123800_bib32","article-title":"Inherently explainable reinforcement learning in natural language","volume":"abs\/2112.08907","author":"Peng","year":"2021","journal-title":"ArXiv"},{"key":"2022082617253123800_bib33","doi-asserted-by":"publisher","first-page":"1135","DOI":"10.1145\/2939672.2939778","article-title":"\u201cwhy should I trust you?\u201d: Explaining the predictions of any classifier","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016","author":"Ribeiro","year":"2016"},{"key":"2022082617253123800_bib34","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1007\/978-3-319-93417-4_38","article-title":"Modeling relational data with graph convolutional networks","volume-title":"European Semantic Web Conference","author":"Schlichtkrull","year":"2018"},{"key":"2022082617253123800_bib35","article-title":"Alfworld: Aligning text and embodied environments for interactive learning","author":"Shridhar","year":"2020","journal-title":"ArXiv"},{"key":"2022082617253123800_bib36","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"2022082617253123800_bib37","volume-title":"Introduction to Reinforcement Learning","author":"Sutton","year":"1998","edition":"1st edition"},{"key":"2022082617253123800_bib38","article-title":"Towards solving text-based games by producing adaptive action spaces","author":"Tao","year":"2018","journal-title":"ArXiv"},{"key":"2022082617253123800_bib39","article-title":"First textworld problems, the competition: Using text-based games to advance capabilities of ai agents","author":"Trischler","year":"2019"},{"key":"2022082617253123800_bib40","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1062","article-title":"Learning to speak and act in a fantasy text adventure game","author":"Urbanek","year":"2019","journal-title":"ArXiv"},{"key":"2022082617253123800_bib41","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022082617253123800_bib42","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1109\/CoG47356.2020.9231622","article-title":"Deep reinforcement learning with transformers for text adventure games","volume-title":"2020 IEEE Conference on Games (CoG)","author":"Xu","year":"2020"},{"key":"2022082617253123800_bib43","article-title":"Deep reinforcement learning with stacked hierarchical attention for text-based games","volume":"33","author":"Yunqiu","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022082617253123800_bib44","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.emnlp-main.704","article-title":"Keep calm and explore: Language models for action generation in text-based games","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Yao","year":"2020"},{"key":"2022082617253123800_bib45","article-title":"Zero-shot learning of text adventure games with sentence-level semantics","author":"Yin","year":"2020","journal-title":"arXiv preprint arXiv:2004.02986"},{"key":"2022082617253123800_bib46","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.273","article-title":"Learning to generalize for sequential decision making","author":"Yin","year":"2020","journal-title":"EMNLP"},{"key":"2022082617253123800_bib47","doi-asserted-by":"publisher","first-page":"2796","DOI":"10.18653\/v1\/D19-1280","article-title":"Interactive language learning by question answering","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Join Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Yuan","year":"2019"},{"key":"2022082617253123800_bib48","article-title":"Counting to explore and generalize in text-based games","volume-title":"35th International Conference on Machine Learning, Exploration in Reinforcement Learning Workshop","author":"Yuan","year":"2018"},{"key":"2022082617253123800_bib49","first-page":"3562","article-title":"Learn what not to learn: Action elimination with deep reinforcement learning","volume-title":"Advances in Neural Information Processing Systems 2018","author":"Zahavy","year":"2018"},{"key":"2022082617253123800_bib50","article-title":"Building dynamic knowledge graphs from text- based games","author":"Zelinka","year":"2019","journal-title":"ArXiv"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00495\/2040849\/tacl_a_00495.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00495\/2040849\/tacl_a_00495.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,26]],"date-time":"2022-08-26T17:26:01Z","timestamp":1661534761000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00495\/112801\/A-Survey-of-Text-Games-for-Reinforcement-Learning"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00495","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}