{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T20:24:50Z","timestamp":1783628690918,"version":"3.55.0"},"reference-count":59,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,15]],"date-time":"2023-08-15T00:00:00Z","timestamp":1692057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["CH2676\/1-1"],"award-info":[{"award-number":["CH2676\/1-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Long-horizon task planning is essential for the development of intelligent assistive and service robots. In this work, we investigate the applicability of a smaller class of large language models (LLMs), specifically GPT-2, in robotic task planning by learning to decompose tasks into subgoal specifications for a planner to execute sequentially. Our method grounds the input of the LLM on the domain that is represented as a scene graph, enabling it to translate human requests into executable robot plans, thereby learning to reason over long-horizon tasks, as encountered in the ALFRED benchmark. We compare our approach with classical planning and baseline methods to examine the applicability and generalizability of LLM-based planners. Our findings suggest that the knowledge stored in an LLM can be effectively grounded to perform long-horizon task planning, demonstrating the promising potential for the future application of neuro-symbolic planning methods in robotics.<\/jats:p>","DOI":"10.3389\/frobt.2023.1221739","type":"journal-article","created":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T09:28:58Z","timestamp":1692178138000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Learning to reason over scene graphs: a case study of finetuning GPT-2 into a robot language model for grounded task planning"],"prefix":"10.3389","volume":"10","author":[{"given":"Georgia","family":"Chalvatzaki","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Younes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daljeet","family":"Nandha","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"An Thai","family":"Le","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leonardo F. R.","family":"Ribeiro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iryna","family":"Gurevych","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,8,15]]},"reference":[{"key":"B1","article-title":"Pddl\u2013 the planning domain definition language","author":"Aeronautiques","year":"1998","journal-title":"Tech. Rep. Tech. Rep."},{"key":"B2","article-title":"A neural probabilistic language model","volume":"13","author":"Bengio","year":"2000","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B3","unstructured":"Chatgpt is a knowledgeable but inexperienced solver: an investigation of commonsense problem in large language models\n            BianN.\n            HanX.\n            SunL.\n            LinH.\n            LuY.\n            HeB.\n          2023"},{"key":"B4","unstructured":"On the opportunities and risks of foundation models\n            BommasaniR.\n            HudsonD. A.\n            AdeliE.\n            AltmanR.\n            AroraS.\n            von ArxS.\n          2021"},{"key":"B5","unstructured":"Do as i can, not as i say: grounding language in robotic affordances\n            BrohanA.\n            ChebotarY.\n            FinnC.\n            HausmanK.\n            HerzogA.\n            HoD.\n          2022"},{"key":"B6","unstructured":"Language models are few-shot learners\n            BrownT.\n            MannB.\n            RyderN.\n            SubbiahM.\n            KaplanJ. D.\n            DhariwalP.\n          2020"},{"key":"B7","unstructured":"Open-vocabulary queryable scene representations for real world planning\n            ChenB.\n            XiaF.\n            IchterB.\n            RaoK.\n            GopalakrishnanK.\n            RyooM. S."},{"key":"B8","unstructured":"Leveraging large language models for robot 3d scene understanding\n            ChenW.\n            HuS.\n            TalakR.\n            CarloneL."},{"key":"B9","unstructured":"Deep visual reasoning: learning to predict action sequences for task and motion planning from an initial scene image\n            DriessD.\n            HaJ.-S.\n            ToussaintM.\n          2020"},{"key":"B10","volume-title":"Hierarchical task and motion planning using logic-geometric programming (hlgp)","author":"Driess","year":"2019"},{"key":"B11","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.proeng.2014.12.098","article-title":"Path planning with modified a star algorithm for a mobile robot","volume":"96","author":"Ducho\u0148","year":"2014","journal-title":"Procedia Eng."},{"key":"B12","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","article-title":"Gpt-3: its nature, scope, limits, and consequences","volume":"30","author":"Floridi","year":"2020","journal-title":"Minds Mach."},{"key":"B13","article-title":"Learn2assemble with structured representations and search for robotic architectural construction","author":"Funk","year":"2021","journal-title":"Conf. Robot Learn. (CoRL)"},{"key":"B14","unstructured":"Graph-based reinforcement learning meets mixed integer programs: an application to 3d robot assembly discovery\n            FunkN.\n            MenzenbachS.\n            ChalvatzakiG.\n            PetersJ.\n          2022"},{"key":"B15","unstructured":"Integrated task and motion planning\n            GarrettC. R.\n            ChitnisR.\n            HolladayR.\n            KimB.\n            SilverT.\n            KaelblingL. P.\n          2021"},{"key":"B16","unstructured":"Pddlstream: integrating symbolic planners and blackbox samplers via optimistic adaptive planning\n            GarrettC. R.\n            Lozano-P\u00e9rezT.\n            KaelblingL. P.\n          2020"},{"key":"B17","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1613\/jair.1705","article-title":"The fast downward planning system","volume":"26","author":"Helmert","year":"2006","journal-title":"J. Artif. Intell. Res."},{"key":"B18","first-page":"26963","article-title":"Successor feature landmarks for long-horizon goal-conditioned reinforcement learning","volume":"34","author":"Hoang","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B19","unstructured":"The curious case of neural text degeneration\n            HoltzmanA.\n            BuysJ.\n            DuL.\n            ForbesM.\n            ChoiY.\n          2020"},{"key":"B20","unstructured":"Parameter-efficient transfer learning for nlp\n            HoulsbyN.\n            GiurgiuA.\n            JastrzebskiS.\n            MorroneB.\n            De LaroussilheQ.\n            GesmundoA.\n          2019"},{"key":"B21","unstructured":"Visual language maps for robot navigation\n            HuangC.\n            MeesO.\n            ZengA.\n            BurgardW."},{"key":"B22","unstructured":"Language models as zero-shot planners: extracting actionable knowledge for embodied agents\n            HuangW.\n            AbbeelP.\n            PathakD.\n            MordatchI."},{"key":"B23","unstructured":"Inner monologue: embodied reasoning through planning with language models\n            HuangW.\n            XiaF.\n            XiaoT.\n            ChanH.\n            LiangJ.\n            FlorenceP."},{"key":"B24","article-title":"Vima: general robot manipulation with multimodal prompts","author":"Jiang","year":"2022"},{"key":"B25","first-page":"1470","article-title":"Hierarchical task and motion planning in the now","author":"Kaelbling","year":"2011"},{"key":"B26","doi-asserted-by":"publisher","first-page":"793","DOI":"10.1177\/0278364919848837","article-title":"Learning to guide task and motion planning using score-space representation","volume":"38","author":"Kim","year":"2019","journal-title":"IJRR"},{"key":"B27","unstructured":"Adam: A method for stochastic optimization\n            KingmaD. P.\n            BaJ.\n          2014"},{"key":"B28","unstructured":"Ai2-thor: an interactive 3d environment for visual ai\n            KolveE.\n            MottaghiR.\n            HanW.\n            VanderBiltE.\n            WeihsL.\n            HerrastiA.\n          2017"},{"key":"B29","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B30","first-page":"31199","article-title":"Pre-trained language models for interactive decision-making","volume":"35","author":"Li","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B31","unstructured":"Do language models learn commonsense knowledge?\n            LiX. L.\n            KuncoroA.\n            d\u2019AutumeC. d. M.\n            BlunsomP.\n            NematzadehA.\n          2021"},{"key":"B32","unstructured":"Code as policies: language model programs for embodied control\n            LiangJ.\n            HuangW.\n            XiaF.\n            XuP.\n            HausmanK.\n            IchterB.\n          2022"},{"key":"B33","unstructured":"Grounding language with visual affordances over unstructured data\n            MeesO.\n            Borja-DiazJ.\n            BurgardW.\n          2022"},{"key":"B34","unstructured":"Hierarchical foresight: self-supervised learning of long-horizon tasks via visual subgoal generation\n            NairS.\n            FinnC.\n          2019"},{"key":"B35","unstructured":"Training language models to follow instructions with human feedback\n            OuyangL.\n            WuJ.\n            JiangX.\n            AlmeidaD.\n            WainwrightC. L.\n            MishkinP.\n          2022"},{"key":"B36","first-page":"15942","article-title":"Episodic transformer for vision-and-language navigation","author":"Pashevich","year":"2021"},{"key":"B37","unstructured":"Adapterfusion: non-destructive task composition for transfer learning\n            PfeifferJ.\n            KamathA.\n            R\u00fcckl\u00e9A.\n            ChoK.\n            GurevychI.\n          2021"},{"key":"B38","unstructured":"\u201cLearning transferable visual models from natural language supervision87488763\n            RadfordA.\n            KimJ. W.\n            HallacyC.\n            RameshA.\n            GohG.\n            AgarwalS.\n          PMLR2021"},{"key":"B39","unstructured":"Planning with large language models via corrective re-prompting\n            RamanS. S.\n            CohenV.\n            RosenE.\n            IdreesI.\n            PauliusD.\n            TellexS.\n          2022"},{"key":"B40","unstructured":"Extended task and motion planning of long-horizon robot manipulation\n            RenT.\n            ChalvatzakiG.\n            PetersJ.\n          2021"},{"key":"B41","unstructured":"Large language models are not zero-shot communicators\n            RuisL.\n            KhanA.\n            BidermanS.\n            HookerS.\n            Rockt\u00e4schelT.\n            GrefenstetteE.\n          2022"},{"key":"B42","unstructured":"Lm-nav: robotic navigation with large pre-trained models of language, vision, and action492504\n            ShahD.\n            Osi\u0144skiB.\n            LevineS.\n          PMLR2023"},{"key":"B43","unstructured":"Alfred: A benchmark for interpreting grounded instructions for everyday tasks\n            ShridharM.\n            ThomasonJ.\n            GordonD.\n            BiskY.\n            HanW.\n            MottaghiR.\n          2020"},{"key":"B44","unstructured":"Progprompt: generating situated robot task plans using large language models\n            SinghI.\n            BlukisV.\n            MousavianA.\n            GoyalA.\n            XuD.\n            TremblayJ.\n          2022"},{"key":"B45","unstructured":"Transcending scaling laws with 0.1% extra compute\n            TayY.\n            WeiJ.\n            ChungH. W.\n            TranV. Q.\n            SoD. R.\n            ShakeriS.\n          2022"},{"key":"B46","article-title":"Logic-geometric programming: an optimization-based approach to combined task and motion planning","author":"Toussaint","year":"2015"},{"key":"B47","unstructured":"Large language models still can\u2019t plan (a benchmark for llms on planning and reasoning about change)\n            ValmeekamK.\n            OlmoA.\n            SreedharanS.\n            KambhampatiS.\n          2022"},{"key":"B48","unstructured":"Attention is all you need\n            VaswaniA.\n            ShazeerN.\n            ParmarN.\n            UszkoreitJ.\n            JonesL.\n            GomezA. N.\n          2017"},{"key":"B49","article-title":"Superglue: A stickier benchmark for general-purpose language understanding systems","volume":"32","author":"Wang","year":"2019","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B50","unstructured":"Glue: A multi-task benchmark and analysis platform for natural language understanding\n            WangA.\n            SinghA.\n            MichaelJ.\n            HillF.\n            LevyO.\n            BowmanS. R.\n          2018"},{"key":"B51","unstructured":"Emergent abilities of large language models\n            WeiJ.\n            TayY.\n            BommasaniR.\n            RaffelC.\n            ZophB.\n            BorgeaudS."},{"key":"B52","unstructured":"Chain-of-thought prompting elicits reasoning in large language models\n            WeiJ.\n            WangX.\n            SchuurmansD.\n            BosmaM.\n            XiaF.\n            ChiE. H."},{"key":"B53","doi-asserted-by":"publisher","first-page":"1255","DOI":"10.1109\/lra.2019.2894861","article-title":"Learning feasibility for task and motion planning in tabletop environments","volume":"4","author":"Wells","year":"2019","journal-title":"IEEE Ral."},{"key":"B54","article-title":"Ll-powered autonomous agents","author":"Weng","year":"2023","journal-title":"lilianweng.github.Io."},{"key":"B55","unstructured":"A prompt pattern catalog to enhance prompt engineering with chatgpt\n            WhiteJ.\n            FuQ.\n            HaysS.\n            SandbornM.\n            OleaC.\n            GilbertH.\n          2023"},{"key":"B56","unstructured":"Huggingface\u2019s transformers: state-of-the-art natural language processing\n            WolfT.\n            DebutL.\n            SanhV.\n            ChaumondJ.\n            DelangueC.\n            MoiA.\n          2019"},{"key":"B57","unstructured":"Accelerating integrated task and motion planning with neural feasibility checking\n            XuL.\n            RenT.\n            ChalvatzakiG.\n            PetersJ.\n          2022"},{"key":"B58","unstructured":"Socratic models: composing zero-shot multimodal reasoning with language\n            ZengA.\n            WongA.\n            WelkerS.\n            ChoromanskiK.\n            TombariF.\n            PurohitA.\n          2022"},{"key":"B59","doi-asserted-by":"publisher","first-page":"9733","DOI":"10.1609\/aaai.v34i05.6523","article-title":"Evaluating commonsense in pre-trained language models","volume":"34","author":"Zhou","year":"2020","journal-title":"Proc. AAAI Conf. Artif. Intell."}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1221739\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T09:29:12Z","timestamp":1692178152000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1221739\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,15]]},"references-count":59,"alternative-id":["10.3389\/frobt.2023.1221739"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1221739","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,15]]},"article-number":"1221739"}}