{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T09:04:11Z","timestamp":1789203851911,"version":"build-2803163510"},"reference-count":60,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T00:00:00Z","timestamp":1785974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>\n                    The rapid advancement of Large Language Models (LLMs) has led to significant progress in Artificial Intelligence (AI), ushering in a new era of human-centered AI (HAI). Intelligent agents powered by LLMs provide new opportunities for realizing HAI. However, existing LLM-based embodied agents often lack online planning capabilities and may generate actions involving objects that are not present in the current environment. In this paper, we propose a closed-loop framework for planning and evaluation of a vision-language model-based robotic manipulation agent operating in tabletop object rearrangement and manipulation tasks. These tasks require a robot to interpret high-level natural language commands and perform multi-step actions such as stacking objects, sorting objects by category or attribute, placing objects into target containers, and organizing mixed desktop scenes. The proposed architecture consists of a human\u2013robot speech interaction module, a vision\u2013language agent module (including a planner, translator, and evaluator), and an action execution module. Within this framework, the vision\u2013language planner decomposes high-level instructions into executable task steps via Chain-of-Objects Reasoning, enabling a deeper understanding of the visual environment, including spatial relationships and object attributes. Meanwhile, the task evaluator monitors execution outcomes and provides result-oriented feedback, allowing the system to replan when errors occur. Experimental results show that, compared with baseline methods, the proposed agent improves the average task success rate by approximately 20% in both simulation and real-world environments, significantly enhancing its ability to execute complex natural language instructions. This work demonstrates the potential of closed-loop vision-language planning for human-centered robotic manipulation. Code:\n                    <jats:ext-link>https:\/\/github.com\/subforpaper\/PFEA<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.3389\/frobt.2026.1905578","type":"journal-article","created":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T04:24:54Z","timestamp":1785990294000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["PFEA: a VLM-based high-level natural language planning and feedback embodied agent for human-centered AI"],"prefix":"10.3389","volume":"13","author":[{"given":"Wenbin","family":"Ding","sequence":"first","affiliation":[{"name":"School of Electrical and Automation Engineering, Nanjing Normal University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Electrical and Automation Engineering, Nanjing Normal University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingjia","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Electrical and Automation Engineering, Nanjing Normal University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fei","family":"Xie","sequence":"additional","affiliation":[{"name":"School of Electrical and Automation Engineering, Nanjing Normal University","place":["Nanjing, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Wang","sequence":"additional","affiliation":[{"name":"Wenzhou Key Laboratory of AI Agents for Agriculture","place":["Wenzhou, Zhejiang, China"]},{"name":"Wenzhou Vocational College of Science and Technology","place":["Wenzhou, Zhejiang, China"]},{"name":"Key Laboratory of Knowledge Engineering with Big Data (Hefei University of Technology), Ministry of Education","place":["Hefei, Anhui, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon X.","family":"Yang","sequence":"additional","affiliation":[{"name":"Advanced Robotics and Intelligent Systems Laboratory, College of Engineering, University of Guelph","place":["Guelph, ON, Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philip","family":"Dames","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Temple University","place":["Philadelphia, PA, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,8,6]]},"reference":[{"key":"B1","article-title":"Mallvi: a multi agent framework for integrated generalized robotics manipulation","author":"Ahmadi","year":"2026"},{"key":"B2","article-title":"Do as i can, not as i say: grounding language in robotic affordances","author":"Ahn","year":"2022"},{"key":"B3","article-title":"Funaudiollm: voice understanding and generation foundation models for natural interaction between humans and llms","author":"An","year":"2024"},{"key":"B4","article-title":"Qwen3-vl technical report","author":"Bai","year":"2025"},{"key":"B5","article-title":"Grounding llms for robot task planning using closed-loop state feedback","author":"Bhat","year":"2024"},{"key":"B6","article-title":"Rt-1: robotics transformer for real-world control at scale","author":"Brohan","year":"2022"},{"key":"B7","article-title":"Rt-2: vision-language-action models transfer web knowledge to robotics control","author":"Brohan","year":"2023"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Information Processing Systems"},{"key":"B9","article-title":"Spatialbot: precise spatial understanding with vision language models","author":"Cai","year":"2024"},{"key":"B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3544548.3580959","article-title":"What is human-centered about human-centered ai? a map of the research landscape","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems","author":"Capel","year":"2023"},{"key":"B11","doi-asserted-by":"publisher","first-page":"4083","DOI":"10.1109\/lra.2023.3346800","article-title":"Can an embodied agent find your \u201ccat-shaped mug\u201d? Llm-based zero-shot object navigation","volume":"9","author":"Dorbala","year":"2023","journal-title":"IEEE Robotics Automation Lett."},{"key":"B12","article-title":"Chatglm: a family of large language models from glm-130b to glm-4 all tools","author":"Glm","year":"2024"},{"key":"B13","article-title":"Metagpt: meta programming for multi-agent collaborative framework","author":"Hong","year":"2023"},{"key":"B14","article-title":"Mirror: modular internal processing for personalized safety in llm dialogue","author":"Hsing","year":"2025"},{"key":"B15","article-title":"Look before you leap: unveiling the power of gpt-4v in robotic vision-language planning","author":"Hu","year":"2023"},{"key":"B16","first-page":"9118","article-title":"Language models as zero-shot planners: extracting actionable knowledge for embodied agents","volume-title":"International Conference on Machine Learning","author":"Huang","year":""},{"key":"B17","article-title":"Inner monologue: embodied reasoning through planning with language models","author":"Huang","year":""},{"key":"B18","doi-asserted-by":"publisher","first-page":"59636","DOI":"10.48550\/arXiv.2303.00855","article-title":"Grounded decoding: guiding text generation with grounded models for embodied agents","volume":"36","author":"Huang","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B19","article-title":"Rekep: spatio-temporal reasoning of relational keypoint constraints for robotic manipulation","author":"Huang","year":"2024"},{"key":"B20","doi-asserted-by":"publisher","first-page":"105197","DOI":"10.1016\/j.robot.2025.105197","article-title":"Integrative ai framework for robotics: Llm-enabled reinforcement learning in object manipulation and task planning","volume":"195","author":"Huynh","year":"2025","journal-title":"Robotics Aut. Syst."},{"key":"B21","article-title":"\u03c0 0.6: a vla that learns from experience","author":"Intelligence","year":""},{"key":"B22","article-title":"\u03c00.5: a vision-language-action model with open-world generalization","author":"Intelligence","year":""},{"key":"B23","article-title":"Mega-tts 2: boosting prompting mechanisms for zero-shot speech synthesis","author":"Jiang","year":"2023"},{"key":"B24","article-title":"Openvla: an open-source vision-language-action model","author":"Kim","year":"2024"},{"key":"B25","doi-asserted-by":"crossref","first-page":"11017","DOI":"10.1109\/ICRA55743.2025.11128028","article-title":"Self-corrective task planning by inverse prompting with large language models","volume-title":"2025 IEEE International Conference on Robotics and Automation (ICRA)","author":"Lee","year":"2025"},{"key":"B26","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1038\/s44387-025-00045-3","article-title":"Self-reflection enhances large language models towards substantial academic response","volume":"1","author":"Li","year":"2025","journal-title":"Artif. Intell."},{"key":"B27","article-title":"How well can llms provide planning feedback in grounded environments?","author":"Li","year":"2025"},{"key":"B28","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1109\/IHCIT66787.2025.11198926","article-title":"Multi-agent collaborative closed-loop planning-evaluation-assessment framework: optimization and interpretability evaluation","volume-title":"2025 4th International Conference on Intelligent Mechanical and Human-Computer Interaction Technology (IHCIT)","author":"Li","year":"2025"},{"key":"B29","doi-asserted-by":"crossref","first-page":"9493","DOI":"10.1109\/ICRA48891.2023.10160591","article-title":"Code as policies: language model programs for embodied control","volume-title":"2023 IEEE International Conference on Robotics and Automation (ICRA)","author":"Liang","year":"2023"},{"key":"B30","doi-asserted-by":"crossref","DOI":"10.15607\/RSS.2024.XX.091","article-title":"Ok-robot: what really matters in integrating open-knowledge models for robotics","author":"Liu","year":"2024"},{"key":"B31","first-page":"18081","article-title":"Hierarchical diffusion policy for kinematics-aware multi-task robotic manipulation","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Ma","year":"2024"},{"key":"B32","article-title":"A survey on vision-language-action models for embodied ai","author":"Ma","year":"2024"},{"key":"B33","doi-asserted-by":"publisher","first-page":"46534","DOI":"10.48550\/arXiv.2303.17651","article-title":"Self-refine: iterative refinement with self-feedback","volume":"36","author":"Madaan","year":"2023","journal-title":"Adv. Neural Information Processing Systems"},{"key":"B34","article-title":"Think, act, learn: a framework for autonomous robotic agents using closed-loop large language models","author":"Menon","year":"2025"},{"key":"B35","first-page":"728","article-title":"Simple open-vocabulary object detection","volume-title":"European Conference on Computer Vision","author":"Minderer","year":"2022"},{"key":"B36","doi-asserted-by":"publisher","first-page":"72983","DOI":"10.48550\/arXiv.2306.09683","article-title":"Scaling open-vocabulary object detection","volume":"36","author":"Minderer","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B37","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1038\/s42256-025-01005-x","article-title":"Embodied large language models enable robots to complete complex tasks in unpredictable environments","volume":"7","author":"Mon-Williams","year":"2025","journal-title":"Nat. Mach. Intell."},{"key":"B38","article-title":"Object-centric agentic robot policies","volume-title":"NeurIPS 2025 Workshop on Space in Vision, Language, and Embodied AI","author":"Morin","year":"2025"},{"key":"B39","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.48550\/arXiv.2203.02155","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv. Neural Information Processing Systems"},{"key":"B40","first-page":"8748","article-title":"Learning transferable visual models from natural language supervision","volume-title":"International Conference on Machine Learning","author":"Radford","year":"2021"},{"key":"B41","article-title":"Sayplan: grounding large language models using 3d scene graphs for scalable robot task planning","author":"Rana","year":"2023"},{"key":"B42","article-title":"A generalist agent","author":"Reed","year":"2022"},{"key":"B43","article-title":"Self-reflection in llm agents: effects on problem-solving performance","author":"Renze","year":"2024"},{"key":"B44","article-title":"Large vlm-based vision-language-action models for robotic manipulation: a survey","author":"Shao","year":"2025"},{"key":"B45","doi-asserted-by":"publisher","first-page":"8634","DOI":"10.48550\/arXiv.2303.11366","article-title":"Reflexion: language agents with verbal reinforcement learning","volume":"36","author":"Shinn","year":"2023","journal-title":"Adv. Neural Information Processing Systems"},{"key":"B46","first-page":"894","article-title":"Cliport: what and where pathways for robotic manipulation","volume-title":"Conference on Robot Learning","author":"Shridhar","year":"2022"},{"key":"B47","article-title":"Progprompt: generating situated robot task plans using large language models","author":"Singh","year":"2022"},{"key":"B48","doi-asserted-by":"crossref","first-page":"20386","DOI":"10.1109\/IROS60139.2025.11247340","article-title":"Malmm: multi-agent large language models for zero-shot robotic manipulation","volume-title":"2025 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Singh","year":"2025"},{"key":"B49","unstructured":"Gemini: a family of highly capable multimodal models\n          \n          \n            \n              Team\n              G.\n            \n            \n              Anil\n              R.\n            \n            \n              Borgeaud\n              S.\n            \n            \n              Alayrac\n              J.-B.\n            \n            \n              Yu\n              J.\n            \n            \n              Soricut\n              R.\n            \n          \n          \n          2023"},{"key":"B50","doi-asserted-by":"publisher","first-page":"5998","DOI":"10.48550\/arXiv.1706.03762","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Information Processing Systems"},{"key":"B51","doi-asserted-by":"publisher","first-page":"55682","DOI":"10.1109\/ACCESS.2024.3387941","article-title":"Chatgpt for robotics: design principles and model abilities","volume":"12","author":"Vemprala","year":"2024","journal-title":"IEEE Access"},{"key":"B52","article-title":"Describe, explain, plan and select: interactive planning with large language models enables open-world multi-task agents","author":"Wang","year":"2023"},{"key":"B53","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1016\/j.jmsy.2024.04.020","article-title":"An llm-based vision and language cobot navigation approach for human-centric smart manufacturing","volume":"75","author":"Wang","year":"2024","journal-title":"J. Manuf. Syst."},{"key":"B54","doi-asserted-by":"crossref","first-page":"6527","DOI":"10.1109\/ICRA55743.2025.11128270","article-title":"Chain-of-modality: learning manipulation programs from multimodal human videos with vision-language-models","volume-title":"2025 IEEE International Conference on Robotics and Automation (ICRA)","author":"Wang","year":"2025"},{"key":"B55","doi-asserted-by":"publisher","first-page":"3988","DOI":"10.1109\/LRA.2025.3544909","article-title":"Tinyvla: towards fast, data-efficient vision-language-action models for robotic manipulation","volume":"10","author":"Wen","year":"2025","journal-title":"IEEE Robotics Automation Lett."},{"key":"B56","first-page":"1","article-title":"On decoder-only architecture for speech-to-text and large language model integration","volume-title":"2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)","author":"Wu","year":"2023"},{"key":"B57","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1109\/jas.2025.125327","article-title":"When embodied ai meets industry 5.0: human-centered smart manufacturing","volume":"12","author":"Xu","year":"2025","journal-title":"IEEE\/CAA J. Automatica Sinica"},{"key":"B58","article-title":"The dawn of lmms: preliminary explorations with gpt-4v (ision)","author":"Yang","year":"2023"},{"key":"B59","first-page":"726","article-title":"Transporter networks: rearranging the visual world for robotic manipulation","volume-title":"Conference on Robot Learning","author":"Zeng","year":"2021"},{"key":"B60","first-page":"2165","article-title":"Rt-2: vision-language-action models transfer web knowledge to robotic control","volume-title":"Conference on Robot Learning (PMLR)","author":"Zitkovich","year":"2023"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2026.1905578\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T04:25:02Z","timestamp":1785990302000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2026.1905578\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,6]]},"references-count":60,"alternative-id":["10.3389\/frobt.2026.1905578"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2026.1905578","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,6]]},"article-number":"1905578"}}