{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T16:24:40Z","timestamp":1792513480039,"version":"4.5.0"},"reference-count":69,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003542","name":"Ministerium f\u00fcr Wissenschaft, Forschung und Kunst Baden-W\u00fcrttemberg","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003542","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007569","name":"Carl-Zeiss-Stiftung","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007569","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Natural-language dialog is key for an intuitive human\u2013robot interaction. It can be used not only to express humans\u2019 intents but also to communicate instructions for improvement if a robot does not understand a command correctly. It is of great importance to let robots learn from such interaction experiences in an incremental way to allow them to improve their behaviors or avoid mistakes in the future. In this paper, we propose a system to achieve such incremental learning of complex high-level behavior from natural interaction and demonstrate its implementation on a humanoid robot. Our system deploys large language models (LLMs) for high-level orchestration of the robot\u2019s behavior based on the idea of enabling the LLM to generate Python statements in an interactive console to invoke both robot perception and action. Human instructions, environment observations, and execution results are fed back to the LLM, thus informing the generation of the next statement. Since an LLM can misunderstand (potentially ambiguous) user instructions, we introduce incremental learning from the interaction, which enables the system to learn from its mistakes. For that purpose, the LLM can call another LLM responsible for code-level improvements in the current interaction based on human feedback. Subsequently, we store the improved interaction in the robot\u2019s memory so that it can later be retrieved on semantically similar requests. We integrate the system in the robot cognitive architecture of the humanoid robot ARMAR-6 and evaluate our methods both quantitatively (in simulation) and qualitatively (in simulation and real-world) by demonstrating generalized incrementally learned knowledge.<\/jats:p>","DOI":"10.3389\/frobt.2024.1455375","type":"journal-article","created":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T11:58:38Z","timestamp":1728561518000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Incremental learning of humanoid robot behavior from natural interaction and large language models"],"prefix":"10.3389","volume":"11","author":[{"given":"Leonard","family":"B\u00e4rmann","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rainer","family":"Kartmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabian","family":"Peller-Konrad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan","family":"Niehues","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Waibel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tamim","family":"Asfour","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,10,10]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1007\/978-3-031-49011-8_37","article-title":"Source-code generation using deep learning: a survey","volume-title":"Progress in artificial intelligence","author":"Ahmed","year":"2023"},{"key":"B2","article-title":"Do as i can, not as i say: grounding language in robotic affordances","author":"Ahn","year":"2022","journal-title":"Annu. Conf. Rob. Learn"},{"key":"B3","first-page":"447","article-title":"ARMAR-6: a collaborative humanoid robot for industrial environments","author":"Asfour","year":"2018"},{"key":"B4","first-page":"1","article-title":"Teach robots understanding new object types and attributes through natural language instructions","author":"Bao","year":"2016"},{"key":"B5","first-page":"610","article-title":"On the dangers of stochastic parrots: can language models be too big?","author":"Bender","year":"2021"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.644","article-title":"Autogpt+p: affordance-based task planning using large language models","author":"Birr","year":"2024"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.5555\/3495724.349588","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Int. Conf. Neural Inf. Process. Syst."},{"key":"B8","article-title":"Evaluating large language models trained on code","author":"Chen","year":"2021"},{"key":"B9","doi-asserted-by":"publisher","first-page":"9465","DOI":"10.18653\/v1\/2022.emnlp-main.644","article-title":"Towards teachable reasoning systems: using a dynamic memory of user feedback for continual system improvement","author":"Dalvi Mishra","year":"2022","journal-title":"Conf. Emp. Meth. Nat. Lang. Proc."},{"key":"B10","doi-asserted-by":"publisher","first-page":"82434","DOI":"10.1109\/ACCESS.2022.3196347","article-title":"Code generation using machine learning: a systematic review","volume":"10","author":"Dehaerne","year":"2022","journal-title":"IEEE Access"},{"key":"B11","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1109\/iros.2013.6696345","article-title":"Using semantic fields to model dynamic spatial relations in a robot architecture for natural language instruction of service robots","volume":"32","author":"Fasola","year":"2013","journal-title":"IEEE\/RSJ Int. Conf. Intel. Rob. Syst."},{"key":"B12","doi-asserted-by":"crossref","DOI":"10.1109\/ICRA.2015.7139462","article-title":"Robot programming by demonstration with situated spatial language understanding","author":"Forbes","year":"2015"},{"key":"B13","unstructured":"Code-chat (Google VertexAI)\n          \n          \n          2023"},{"key":"B14","doi-asserted-by":"publisher","first-page":"1640","DOI":"10.1109\/iros.2013.6696569","article-title":"Grounding spatial relations for human-robot interaction","volume":"21","author":"Guadarrama","year":"2013","journal-title":"IEEE\/RSJ Int. Conf. Intel. Rob. Syst."},{"key":"B15","article-title":"Language models can teach themselves to program better","author":"Haluptzok","year":"2023"},{"key":"B16","first-page":"9118","article-title":"Language models as zero-shot planners: extracting actionable knowledge for embodied agents","volume":"162","author":"Huang","year":"","journal-title":"Int. Conf. Mach. Learn."},{"key":"B17","article-title":"Grounded decoding: guiding text generation with grounded models for robot control","author":"Huang","year":"2023"},{"key":"B18","article-title":"Inner monologue: embodied reasoning through planning with language models","author":"Huang","year":""},{"key":"B19","doi-asserted-by":"publisher","first-page":"1151303","DOI":"10.3389\/frobt.2023.1151303","article-title":"Interactive and incremental learning of spatial object relations from human demonstrations","volume":"10","author":"Kartmann","year":"2023","journal-title":"Front. Robotics AI"},{"key":"B20","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1016\/j.robot.2011.05.009","article-title":"Object\u2013Action Complexes: grounded abstractions of sensory\u2013motor processes","volume":"59","author":"Kr\u00fcger","year":"2011","journal-title":"Rob. Auton. Sys."},{"key":"B21","article-title":"Interactive code generation via test-driven user-intent formalization","author":"Lahiri","year":"2023"},{"key":"B22","doi-asserted-by":"publisher","first-page":"21314","DOI":"10.5555\/3600270.3601819","article-title":"CodeRL: mastering code generation through pretrained models and deep reinforcement learning","volume":"35","author":"Le","year":"2022","journal-title":"Int. Conf. Neural Inf. Process. Syst."},{"key":"B23","first-page":"9493","article-title":"Code as policies: language model programs for embodied control","author":"Liang","year":"2023"},{"key":"B24","first-page":"100","article-title":"What makes good in-context examples for GPT-3?","author":"Liu","year":"2022"},{"key":"B25","article-title":"AgentBench: evaluating LLMs as agents","author":"Liu","year":"2024"},{"key":"B26","article-title":"Dr.ICL: demonstration-retrieved in-context learning","author":"Luo","year":"2023"},{"key":"B27","doi-asserted-by":"publisher","first-page":"2833","DOI":"10.18653\/v1\/2022.emnlp-main.183","article-title":"Memory-assisted prompt editing to improve GPT-3 after deployment","author":"Madaan","year":"","journal-title":"Conf. Emp. Meth. Nat. Lang. Proc."},{"key":"B28","doi-asserted-by":"publisher","first-page":"1384","DOI":"10.18653\/v1\/2022.emnlp-main.90","article-title":"Language models of code are few-shot commonsense learners","author":"Madaan","year":"","journal-title":"Conf. Emp. Meth. Nat. Lang. Proc."},{"key":"B29","article-title":"Augmented language models: a survey","author":"Mialon","year":"2023","journal-title":"Trans. Mach. Learn. Res."},{"key":"B30","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1177\/0278364915602060","article-title":"Tell me Dave: context-sensitive grounding of natural language to manipulation instructions","volume":"35","author":"Misra","year":"2016","journal-title":"Int. J. Rob. Res."},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v28i1.8756","article-title":"Learning goal-oriented hierarchical tasks from situated interactive instruction","volume":"28","author":"Mohan","year":"2014","journal-title":"AAAI"},{"key":"B32","first-page":"770","article-title":"Learning of complex-structured tasks from verbal instruction","author":"Nicolescu","year":"2019"},{"key":"B33","article-title":"CodeGen: an open large language model for code with multi-turn program synthesis","author":"Nijkamp","year":"2023"},{"key":"B34","unstructured":"ChatGPT"},{"key":"B35","article-title":"GPT-4 technical report","year":""},{"key":"B36","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.5555\/3600270.3602281","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Int. Conf. Neural Inf. Process. Syst."},{"key":"B37","article-title":"Lifelong robot learning with human assisted language planners","author":"Parakh","year":"2023","journal-title":"Work. learn. Eff. Abstr. Plan. CoRL"},{"key":"B38","article-title":"TALM: tool augmented language models","author":"Parisi","year":"2022"},{"key":"B39","doi-asserted-by":"publisher","first-page":"104415","DOI":"10.1016\/j.robot.2023.104415","article-title":"A memory system of a robot cognitive architecture and its implementation in ArmarX","volume":"164","author":"Peller-Konrad","year":"2023","journal-title":"Rob. Auton. Sys."},{"key":"B40","doi-asserted-by":"crossref","DOI":"10.1109\/IROS45743.2020.9341289","article-title":"DeComplex: task planning from complex natural instructions by a collocating robot","author":"Pramanick","year":"2020"},{"key":"B41","article-title":"Tool learning with foundation models","author":"Qin","year":"2023"},{"key":"B42","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: sentence embeddings using siamese BERT-networks","author":"Reimers","year":"2019","journal-title":"Conf. Emp. Meth. Nat. Lang. Proc."},{"key":"B43","article-title":"Robots that ask for help: uncertainty alignment for large language model planners","author":"Ren","year":"2023"},{"key":"B44","doi-asserted-by":"publisher","first-page":"3468","DOI":"10.18653\/v1\/2023.findings-emnlp.226","article-title":"Open-ended instructable embodied agents with memory-augmented large language models","author":"Sarch","year":"2023","journal-title":"Conf. Emp. Meth. Nat. Lang. Proc."},{"key":"B45","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1177\/0278364919897133","article-title":"INGRESS: interactive visual grounding of referring expressions","volume":"39","author":"Shridhar","year":"2020","journal-title":"Int. J. Rob. Res."},{"key":"B46","first-page":"11523","article-title":"ProgPrompt: generating situated robot task plans using large language models","author":"Singh","year":"2023"},{"key":"B47","article-title":"Errors are useful prompts: instruction guided task programming with verifier-assisted iterative prompting","author":"Skreta","year":"2023"},{"key":"B48","first-page":"2998","article-title":"LLM-planner: few-shot grounded planning for embodied agents with large language models","author":"Song","year":"2023"},{"key":"B49","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1146\/annurev-control-101119-071628","article-title":"Robots that use language: a survey","volume":"3","author":"Tellex","year":"2020","journal-title":"Annu. Rev. Control Rob. Auton. Sys."},{"key":"B50","doi-asserted-by":"publisher","first-page":"1507","DOI":"10.1609\/aaai.v25i1.7979","article-title":"Understanding natural language commands for robotic navigation and mobile manipulation","volume":"25","author":"Tellex","year":"2011","journal-title":"AAAI"},{"key":"B51","article-title":"LLaMA: open and efficient foundation language models","author":"Touvron","year":"2023"},{"key":"B52","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1515\/itit-2014-1066","article-title":"The robot software framework ArmarX","volume":"57","author":"Vahrenkamp","year":"2015","journal-title":"it - Inf. Technol."},{"key":"B53","unstructured":"ChatGPT for robotics: design principles and model abilities\n          \n          \n            \n              Vemprala\n              S.\n            \n            \n              Bonatti\n              R.\n            \n            \n              Bucker\n              A.\n            \n            \n              Kapoor\n              A.\n            \n          \n          \n          2023"},{"key":"B54","doi-asserted-by":"publisher","first-page":"95060","DOI":"10.1109\/access.2023.3310935","article-title":"ChatGPT empowered long-step robot control in various environments: a case application","volume":"11","author":"Wake","year":"2023","journal-title":"IEEE Access"},{"key":"B55","doi-asserted-by":"crossref","DOI":"10.15607\/RSS.2013.IX.004","article-title":"Learning semantic maps from natural language descriptions","author":"Walter","year":"2013"},{"key":"B56","article-title":"Voyager: an open-ended embodied agent with large language models","author":"Wang","year":"2024","journal-title":"Trans. Mach. Learn. Res."},{"key":"B57","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1109\/MedAI59581.2023.00044","article-title":"A review on code generation with LLMs: application and evaluation","volume":"32","author":"Wang","year":"2023","journal-title":"Int. Conf. Med. Art. Intel."},{"key":"B58","article-title":"MINT: evaluating LLMs in multi-turn interaction with tools and language feedback","author":"Wang","year":"2024"},{"key":"B59","article-title":"Interactive natural language processing","author":"Wang","year":"2023"},{"key":"B60","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei","year":"2022"},{"key":"B61","doi-asserted-by":"crossref","first-page":"4280","DOI":"10.18653\/v1\/2020.acl-main.395","article-title":"Programming in natural language with fuSE: synthesizing methods from spoken utterances using deep natural language understanding","volume-title":"Proceedings of the 58th annual meeting of the association for computational linguistics","author":"Weigelt","year":"2020"},{"key":"B62","doi-asserted-by":"publisher","first-page":"1087","DOI":"10.1007\/s10514-023-10139-z","article-title":"TidyBot: personalized robot assistance with large language models","volume":"47","author":"Wu","year":"2023","journal-title":"Auton. Robots"},{"key":"B63","first-page":"23826","article-title":"InterCode: standardizing and benchmarking interactive coding with execution feedback","author":"Yang","year":"2023"},{"key":"B64","article-title":"If LLM is the wizard, then code is the wand: a survey on how code empowers large language models to serve as intelligent agents","author":"Yang","year":"2024"},{"key":"B65","article-title":"ReAct: synergizing reasoning and acting in language models","author":"Yao","year":"2023"},{"key":"B66","article-title":"Compositional exemplars for in-context learning","author":"Ye","year":"2023"},{"key":"B67","article-title":"Socratic models: composing zero-shot multimodal reasoning with language","author":"Zeng","year":"2023"},{"key":"B68","article-title":"Distilling and retrieving generalizable knowledge for robot manipulation via language corrections","volume-title":"Work. Lang. Robot learn., CoRL","author":"Zha","year":"2023"},{"key":"B69","article-title":"A survey of large language models for code: evolution, benchmarking, and future trends","author":"Zheng","year":"2024"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1455375\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T11:58:54Z","timestamp":1728561534000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1455375\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,10]]},"references-count":69,"alternative-id":["10.3389\/frobt.2024.1455375"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2024.1455375","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,10]]},"article-number":"1455375"}}