{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T20:39:10Z","timestamp":1789677550084,"version":"4.0.1"},"reference-count":45,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T00:00:00Z","timestamp":1774051200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Korea government","award":["2022-0-00871"],"award-info":[{"award-number":["2022-0-00871"]}]},{"name":"Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration","award":["RS-2022-00187238"],"award-info":[{"award-number":["RS-2022-00187238"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Embodied artificial intelligence (AI), which integrates AI and robotics, has made significant progress, particularly in human\u2013robot interaction, task-assisting robots, and the integration of multimodal AI models. Experimental studies have demonstrated strong performance in complex tasks, such as providing human assistance, performing household chores, and object manipulation through pick-and-place operations. However, despite these impressive capabilities, real-world applicability remains limited. While tasks such as household chores and object manipulation offer significant practical utility, users often struggle to provide effective instructions, and execution remains prohibitively slow for real-world deployment. This study introduces an approach to enhance usability through spoken human instructions and reduce operation time by streamlining intermediate steps through our Module Handler. The proposed approach leverages a large language model to extract information from spoken human instructions accurately. Through experiments, we validated the accuracy of our approach and confirmed speed improvements compared with related studies. Our experiments evaluated system accuracy in extracting relevant information from spoken human instruction, achieving an object identification accuracy rate of approximately 92.47%. In addition, our method reduced task completion times by an average of 33 s across four different experimental environments compared with existing modular robotics systems. This time reduction is significant for enhancing robotic task execution efficiency.<\/jats:p>","DOI":"10.3390\/s26061978","type":"journal-article","created":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T08:47:42Z","timestamp":1774255662000},"page":"1978","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Streamlining Human\u2013Robot Interaction: Integrating LLM-Based Planning into Modular Robotic Frameworks"],"prefix":"10.3390","volume":"26","author":[{"given":"MinHyuk","family":"Kim","sequence":"first","affiliation":[{"name":"Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"JooHee","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Korea University, Anam-ro 145, Seongbuk-gu, Seoul 02841, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kwanyong","family":"Park","sequence":"additional","affiliation":[{"name":"Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6538-4701","authenticated-orcid":false,"given":"Yong-Ju","family":"Lee","sequence":"additional","affiliation":[{"name":"Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanghun","family":"Jeon","sequence":"additional","affiliation":[{"name":"Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1232","DOI":"10.1080\/01691864.2024.2408593","article-title":"Real-world robot applications of foundation models: A review","volume":"38","author":"Kawaharazuka","year":"2024","journal-title":"Adv. Robot."},{"key":"ref_2","first-page":"35","article-title":"Artificial intelligence, machine learning and deep learning in advanced robotics, a review","volume":"3","author":"Tulli","year":"2024","journal-title":"Int. J. Acta Inform."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"850","DOI":"10.1080\/01691864.2022.2114297","article-title":"World robot challenge 2020\u2014Partner robot: A data-driven approach for room tidying with mobile manipulator","volume":"36","author":"Matsushima","year":"2022","journal-title":"Adv. Robot."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lin, M.-Y., Lee, O.-W., and Lu, C.-Y. (2024). Embodied AI with large language models: A survey and new HRI framework. Proceedings of the International Conference on Advanced Robotics and Mechatronics (ICARM), Tokyo, Japan, 8\u201310 July 2024, IEEE.","DOI":"10.1109\/ICARM62033.2024.10715872"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1109\/TETCI.2022.3141105","article-title":"A survey of embodied AI: From simulators to research tasks","volume":"6","author":"Duan","year":"2022","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"836","DOI":"10.4218\/etrij.2023-0116","article-title":"Real-time 3D multi-pedestrian detection and tracking using 3D LIDAR point cloud for mobile robot","volume":"45","author":"Na","year":"2023","journal-title":"ETRI J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"71","DOI":"10.4218\/etrij.2023-0352","article-title":"KMSAV: Korean multi-speaker spontaneous audiovisual dataset","volume":"46","author":"Park","year":"2024","journal-title":"ETRI J."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"329","DOI":"10.4218\/etrij.2021-0397","article-title":"A robust collision prediction and detection method based on neural network for autonomous delivery robots","volume":"45","author":"Seo","year":"2023","journal-title":"ETRI J."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"511","DOI":"10.4218\/etrij.2018-0057","article-title":"Work chain-based inverse kinematics of robot to imitate human motion with Kinect","volume":"40","author":"Zhang","year":"2018","journal-title":"ETRI J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"617","DOI":"10.4218\/etrij.2021-0061","article-title":"Onboard dynamic RGB-D simultaneous localization and mapping for mobile robot navigation","volume":"43","author":"Canovas","year":"2021","journal-title":"ETRI J."},{"key":"ref_11","unstructured":"OpenAI (2023, September 12). GPT-4, 2023. OpenAI. Available online: https:\/\/openai.com\/gpt-4."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. (2023). Code as policies: Language model programs for embodied control. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 29 May\u20132 June 2023, IEEE.","DOI":"10.1109\/ICRA48891.2023.10160591"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"55682","DOI":"10.1109\/ACCESS.2024.3387941","article-title":"ChatGPT for robotics: Design principles and model abilities","volume":"12","author":"Vemprala","year":"2024","journal-title":"IEEE Access"},{"key":"ref_14","first-page":"100428","article-title":"Embodied agent interface: Benchmarking LLMs for embodied decision making","volume":"37","author":"Li","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Dorbala, V.S., Mullen, J.F., and Manocha, D. (2023). Can an embodied agent find your Cat-Shaped Mug? LLM-guided exploration for zero-shot object navigation. arXiv.","DOI":"10.1109\/LRA.2023.3346800"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"6728","DOI":"10.1109\/LRA.2024.3410155","article-title":"Language models as zero-shot trajectory generators","volume":"9","author":"Kwon","year":"2024","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_17","unstructured":"Yoshida, T., Masumori, A., and Ikegami, T. (2023). From text to motion: Grounding GPT-4 in a humanoid robot \u201calter3\u201d. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dai, Y., Peng, R., Li, S., and Chai, J. (2024). Think, act, and ask: Open-world interactive personalized robot navigation. arXiv.","DOI":"10.1109\/ICRA57147.2024.10610178"},{"key":"ref_19","unstructured":"Ma, Y., Song, Z., Zhuang, Y., Hao, J., and King, I. (2024). A survey on vision-language-action models for embodied AI. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1109\/JAS.2025.125327","article-title":"When embodied AI meets Industry 5.0: Human-centered smart manufacturing","volume":"12","author":"Xu","year":"2025","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"12719","DOI":"10.1007\/s10489-022-04148-1","article-title":"Sim-to-real transfer and reality gap modeling in model predictive control for autonomous driving","volume":"53","author":"Daza","year":"2023","journal-title":"Appl. Intell."},{"key":"ref_22","first-page":"734","article-title":"What went wrong? Closing the sim-to-real gap via differentiable causal discovery","volume":"229","author":"Huang","year":"2023","journal-title":"Proc. Mach. Learn. Res."},{"key":"ref_23","unstructured":"Liu, V., Adeniji, A., Zhan, H., Haldar, S., Bhirangi, R., Abbeel, P., and Pinto, L. (2025). EgoZero: Robot learning from smart glasses. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1597","DOI":"10.1007\/s12555-025-0127-1","article-title":"Reinforced intelligence through active interaction in real world: A survey on embodied AI","volume":"23","author":"Kwon","year":"2025","journal-title":"Int. J. Control Autom. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"22","DOI":"10.4218\/etrij.2023-0266","article-title":"Multimodal audiovisual speech recognition architecture using a three-feature multi-fusion method for noise-robust systems","volume":"46","author":"Jeon","year":"2024","journal-title":"ETRI J."},{"key":"ref_26","first-page":"25081","article-title":"EmbodiedGPT: Vision-language pre-training via embodied chain of thought","volume":"36","author":"Mu","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","unstructured":"Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J.B., Shu, T., and Gan, C. (2023). Building cooperative embodied agents modularly with large language models. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.cogr.2023.04.001","article-title":"Artificial intelligence, machine learning and deep learning in advanced robotics, a review","volume":"3","author":"Soori","year":"2023","journal-title":"Cogn. Robot."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1146\/annurev-control-042920-020211","article-title":"Safe learning in robotics: From learning-based control to safe reinforcement learning","volume":"5","author":"Brunke","year":"2022","journal-title":"Annu. Rev. Control Robot. Auton. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e36671","DOI":"10.1097\/MD.0000000000036671","article-title":"Ethical implications of AI and robotics in healthcare: A review","volume":"102","author":"Elendu","year":"2023","journal-title":"Medicine"},{"key":"ref_31","first-page":"24","article-title":"A review of the literature on the subject of ethical and risk considerations in the context of fast AI development","volume":"5","author":"Pasham","year":"2022","journal-title":"Int. J. Mod. Comput."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zheng, D., Huang, S., Zhao, L., Zhong, Y., and Wang, L. (2024). Towards learning a generalist model for embodied navigation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, IEEE.","DOI":"10.1109\/CVPR52733.2024.01293"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1007\/s10514-022-10039-8","article-title":"Motion planning and control for mobile robot navigation using machine learning: A survey","volume":"46","author":"Xiao","year":"2022","journal-title":"Auton. Robot."},{"key":"ref_34","unstructured":"Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). Scalable deep reinforcement learning for vision-based robotic manipulation. Proceedings of the 2nd Conference on Robot Learning, PMLR."},{"key":"ref_35","unstructured":"Shafiullah, N.M.M., Paxton, C., Pinto, L., Chintala, S., and Szlam, A. (2023). CLIP-Fields: Weakly supervised semantic fields for robotic memory. arXiv."},{"key":"ref_36","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021). Learning transferable visual models from natural language supervision. arXiv."},{"key":"ref_37","unstructured":"Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. (2024). Multilingual E5 Text Embeddings: A Technical Report. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bhirangi, R., Pattabiraman, V., Erciyes, E., Cao, Y., Hellebrekers, T., and Pinto, L. (2024). AnySkin: Plug-and-Play Skin Sensing for Robotic Touch. arXiv.","DOI":"10.1109\/ICRA55743.2025.11128638"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Liu, P., Orru, Y., Vakil, J., Paxton, C., Shafiullah, N.M.M., and Pinto, L. (2024). OK-Robot: What really matters in integrating open-knowledge models for robotics. arXiv.","DOI":"10.15607\/RSS.2024.XX.091"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023). Segment anything. Proceedings of the IEEE\/CVF International Conference on Computer Vision, IEEE.","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"3929","DOI":"10.1109\/TRO.2023.3281153","article-title":"AnyGrasp: Robust and efficient grasp perception in spatial and temporal domains","volume":"39","author":"Fang","year":"2023","journal-title":"IEEE Trans. Robot."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., and Shen, Z. (2022). Simple open-vocabulary object detection with vision transformers. arXiv.","DOI":"10.1007\/978-3-031-20080-9_42"},{"key":"ref_43","unstructured":"Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zhu, Q.-S., Zhou, L., Zhang, J., Liu, S.-J., Hu, Y.-C., and Dai, L.-R. (2022). Robust data2vec: Noise-robust speech representation learning for ASR by combining regression and improved contrastive learning. arXiv.","DOI":"10.1109\/ICASSP49357.2023.10095373"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3495246","article-title":"Have I Got the Power? Analysing and Reporting Statistical Power in HRI","volume":"11","author":"Bartlett","year":"2022","journal-title":"J. Hum.-Robot Interact."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/6\/1978\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T08:51:00Z","timestamp":1774255860000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/6\/1978"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,21]]},"references-count":45,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["s26061978"],"URL":"https:\/\/doi.org\/10.3390\/s26061978","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,21]]}}}