{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T13:35:26Z","timestamp":1777988126568,"version":"3.51.4"},"reference-count":45,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T00:00:00Z","timestamp":1773964800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"crossref","award":["2138514"],"award-info":[{"award-number":["2138514"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000001","name":"U.S. National Science Foundation","doi-asserted-by":"crossref","award":["2222670"],"award-info":[{"award-number":["2222670"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Widespread acceptance of collaborative robots in human-involved scenarios requires accessible and intuitive interfaces for lay workers and non-expert users. Existing interfaces often rely on users to plan and issue low-level commands, necessitating extensive knowledge of robot control. This study proposes a multimodal agentic AI framework integrating natural user interfaces (NUIs) to foster effortless human-like partnerships in human\u2013robot collaboration (HRC), which enhance intuitiveness and operational efficiency. First, it allows users to instruct robots using plain language verbally, coupled with gaze, revealing objects precisely. Second, it offloads users\u2019 workload for robot motion planning by understanding context and reasoning task decomposition. Third, coordinating with AI agents built on large language models (LLMs), the system interprets users\u2019 requests effectively and provides feedback to establish transparent communication. This proof-of-concept study included experiments to demonstrate a practical implementation of the agentic AI framework on a mobile manipulation robot in the collaborative task of human\u2013robot wood assembly. Seven participants were recruited to interact with this AI-integrated agentic robotic system. Task performance and user experience metrics were measured in terms of completion time, intervention rate, NASA TLX survey for workload, and valuable insights of practical applications were summarized through a qualitative analysis. This study highlights the potential of NUIs and agentic AI-embodied robots to overcome existing HRC barriers and contributes to improving HRC intuitiveness and efficiency.<\/jats:p>","DOI":"10.3390\/s26061958","type":"journal-article","created":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T17:02:43Z","timestamp":1774026163000},"page":"1958","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A Multimodal Agentic AI Framework for Intuitive Human\u2013Robot Collaboration"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4149-9154","authenticated-orcid":false,"given":"Xiaoyun","family":"Liang","sequence":"first","affiliation":[{"name":"School of Civil & Environmental Engineering, and Construction Management, The University of Texas at San Antonio, San Antonio, TX 78249, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6110-5293","authenticated-orcid":false,"given":"Jiannan","family":"Cai","sequence":"additional","affiliation":[{"name":"School of Civil & Environmental Engineering, and Construction Management, The University of Texas at San Antonio, San Antonio, TX 78249, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"745","DOI":"10.1007\/s12369-023-00977-3","article-title":"A systematic literature review of decision-making and control systems for autonomous and social robots","volume":"15","author":"Malfaz","year":"2023","journal-title":"Int. J. Soc. Robot."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"103102","DOI":"10.1016\/j.rcim.2025.103102","article-title":"Exploring tasks and challenges in human-robot collaborative systems: A review","volume":"97","author":"Asif","year":"2026","journal-title":"Robot. Comput.-Integr. Manuf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1080\/00051144.2011.11828422","article-title":"Dynamaid, an anthropomorphic robot for research on domestic service applications","volume":"52","author":"Behnke","year":"2011","journal-title":"Automatika"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.autcon.2019.02.003","article-title":"Tablet interface for direct vision teleoperation of an excavator for urban construction work","volume":"102","author":"Okishiba","year":"2019","journal-title":"Autom. Constr."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/s43154-020-00005-6","article-title":"Review of interfaces for industrial human-robot interaction","volume":"1","author":"Berg","year":"2020","journal-title":"Curr. Robot. Rep."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Benos, L., Moysiadis, V., Kateris, D., Tagarakis, A.C., Busato, P., Pearson, S., and Bochtis, D. (2023). Human\u2013robot interaction in agriculture: A systematic review. Sensors, 23.","DOI":"10.3390\/s23156776"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Jain, J., Lund, A., and Wixon, D. (2011). The future of natural user interfaces. Proceedings of the CHI \u201911 Extended Abstracts on Human Factors in Computing Systems, Vancouver, BC, Canada, 7\u201312 May 2011, Association for Computing Machinery.","DOI":"10.1145\/1979742.1979527"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"101830","DOI":"10.1016\/j.rcim.2019.101830","article-title":"Bi-directional navigation intent communication using spatial augmented reality and eye-tracking glasses for improved safety in human\u2013robot interaction","volume":"61","author":"Chadalavada","year":"2020","journal-title":"Robot. Comput.-Integr. Manuf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"55748","DOI":"10.1109\/ACCESS.2023.3282111","article-title":"Improved trust in human-robot collaboration with ChatGPT","volume":"11","author":"Ye","year":"2023","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"102572","DOI":"10.1016\/j.aei.2024.102572","article-title":"Embodied AI for dexterity-capable construction Robots: DEXBOT framework","volume":"62","author":"You","year":"2024","journal-title":"Adv. Eng. Inform."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"116218","DOI":"10.1016\/j.rser.2025.116218","article-title":"Beyond automation: Unveiling the potential of agentic intelligence","volume":"226","author":"Tirulo","year":"2026","journal-title":"Renew. Sustain. Energy Rev."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Anjomshoae, S., Najjar, A., Calvaresi, D., and Fr\u00e4mling, K. (2019). Explainable Agents and Robots: Results from a Systematic Literature Review. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, Montreal, QC, Canada, 13\u201317 May 2019, International Foundation for Autonomous Agents and Multiagent Systems.","DOI":"10.65109\/KCZB5817"},{"key":"ref_13","unstructured":"Newn, J., Singh, R., Allison, F., Madumal, P., Velloso, E., and Vetere, F. (2019). Designing Interactions with Intention-Aware Gaze-Enabled Artificial Agents. Proceedings of the Human-Computer Interaction\u2014INTERACT 2019: 17th IFIP TC 13 International Conference, Paphos, Cyprus, 2\u20136 September 2019, Springer."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1007\/s11370-026-00700-x","article-title":"Variable admittance control for door opening with a wheeled mobile manipulator considering ground obstacles","volume":"19","author":"Xing","year":"2026","journal-title":"Intell. Serv. Robot."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Angrisani, L., Arpaia, P., Moccaldi, N., and Esposito, A. (2018). Wearable Augmented Reality and Brain Computer Interface to Improve Human-Robot Interactions in Smart Industry: A Feasibility Study for SSVEP Signals, IEEE.","DOI":"10.1109\/RTSI.2018.8548517"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"03121006","DOI":"10.1061\/(ASCE)CO.1943-7862.0002154","article-title":"Human\u2013robot collaboration in construction: Classification and research trends","volume":"147","author":"Liang","year":"2021","journal-title":"J. Constr. Eng. Manag."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"04020002","DOI":"10.1061\/(ASCE)SC.1943-5576.0000468","article-title":"Utilizing emerging technologies for construction safety risk mitigation","volume":"25","author":"Okpala","year":"2020","journal-title":"Pract. Period. Struct. Des. Constr."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1016\/j.procir.2019.02.080","article-title":"Human-Robot-Interaction for mobile industrial robot teams","volume":"79","author":"Berg","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.procir.2018.03.156","article-title":"Human-robot collaboration\u2013towards new metrics for selection of communication technologies","volume":"72","author":"Gustavsson","year":"2018","journal-title":"Procedia CIRP"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, D., Goyal, A., Newell, A., Lee, S., Deng, J., and Kamat, V.R. (2019). Semantic Relation Detection Between Construction Entities to Support Safe Human-Robot Collaboration in Construction, American Society of Civil Engineers.","DOI":"10.1061\/9780784482438.034"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"103523","DOI":"10.1016\/j.autcon.2020.103523","article-title":"Brain-computer interface for hands-free teleoperation of construction robots","volume":"123","author":"Liu","year":"2021","journal-title":"Autom. Constr."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1016\/j.autcon.2018.06.014","article-title":"Technology gaps in human-machine interfaces for autonomous construction robots","volume":"94","author":"Czarnowski","year":"2018","journal-title":"Autom. Constr."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Liang, X., Cruz Rivera, J., Cai, J., and Li, S. (2024). Gaze-Enhanced and LLM-Enabled System for Intuitive Human-Robot Collaboration. Computing in Civil Engineering 2024, American Society of Civil Engineers.","DOI":"10.1061\/9780784486115.102"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"126360","DOI":"10.1016\/j.eswa.2024.126360","article-title":"NVP-HRI: Zero shot natural voice and posture-based human\u2013robot interaction via large language model","volume":"268","author":"Lai","year":"2025","journal-title":"Expert Syst. Appl."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Liu, F., Fang, K., Abbeel, P., and Levine, S. (2024). Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv.","DOI":"10.15607\/RSS.2024.XX.062"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"106313","DOI":"10.1016\/j.ssci.2023.106313","article-title":"Occupational health and safety issues in human-robot collaboration: State of the art and open challenges","volume":"169","author":"Giallanza","year":"2024","journal-title":"Saf. Sci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"103591","DOI":"10.1016\/j.aei.2025.103591","article-title":"Contexts Matter: Robot-Aware 3D human motion prediction for Agentic AI-empowered Human-Robot collaboration","volume":"68","author":"Liang","year":"2025","journal-title":"Adv. Eng. Inform."},{"key":"ref_28","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 12\u201317). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"975","DOI":"10.1111\/j.1540-5915.1997.tb01338.x","article-title":"Virtual teams versus face-to-face teams: An exploratory study of a web-based conference system","volume":"28","author":"Warkentin","year":"1997","journal-title":"Decis. Sci."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Fuchs, S., and Belardinelli, A. (2021). Gaze-based intention estimation for shared autonomy in pick-and-place tasks. Front. Neurorobotics, 15.","DOI":"10.3389\/fnbot.2021.647930"},{"key":"ref_31","unstructured":"Picovoice Inc. (2026, January 30). Picovoice: On-device Voice AI and Local LLM Platforms for Enterprises. Available online: https:\/\/picovoice.ai\/."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"73890","DOI":"10.1109\/ACCESS.2021.3080517","article-title":"Towards open and expandable cognitive AI architectures for large-scale multi-agent human-robot collaborative learning","volume":"9","author":"Papadopoulos","year":"2021","journal-title":"IEEE Access"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Liang, X., Cai, J., and Hu, Y. (2024). Analyzing Human Visual Attention in Human-Robot Collaborative Construction Tasks, American Society of Civil Engineers.","DOI":"10.1061\/9780784485262.087"},{"key":"ref_34","unstructured":"Kuykendall, M.A. (2026, January 30). The 5MB Alternative to Ollama. Available online: https:\/\/github.com\/mkll\/shimmy-ai."},{"key":"ref_35","unstructured":"Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., and Vaughan, A. (2024). The llama 3 herd of models. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Nielsen, J., and Landauer, T.K. (1993). A mathematical model of the finding of usability problems. Proceedings of the INTERACT \u201893 and CHI \u201893 Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands, 24\u201329 April 1993, Association for Computing Machinery.","DOI":"10.1145\/169059.169166"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research","volume":"Volume 52","author":"Hart","year":"1988","journal-title":"Advances in Psychology"},{"key":"ref_38","unstructured":"International Code Council (2026, January 01). 2018 International Residential Code (IRC). Available online: https:\/\/codes.iccsafe.org\/content\/IRC2018."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Laumond, J.-P. (1998). Robot Motion Planning and Control, Springer.","DOI":"10.1007\/BFb0036069"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1177\/0278364919868017","article-title":"Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching","volume":"41","author":"Zeng","year":"2022","journal-title":"Int. J. Robot. Res."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kassner, M., Patera, W., and Bulling, A. (2014, January 13\u201317). Pupil: An open source platform for pervasive eye tracking and mobile gaze-based interaction. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, Seattle, WA, USA.","DOI":"10.1145\/2638728.2641695"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., and Goodrich, M. (2006). Common metrics for human-robot interaction. Proceedings of the 1st ACM SIGCHI\/SIGART Conference on HUMAN-Robot Interaction, Salt Lake City, UT, USA, 2\u20133 March 2006, Association for Computing Machinery.","DOI":"10.1145\/1121241.1121249"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"104224","DOI":"10.1016\/j.apergo.2024.104224","article-title":"Exploring the impact of human-robot interaction on workers\u2019 mental stress in collaborative assembly tasks","volume":"116","author":"Su","year":"2024","journal-title":"Appl. Ergon."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Player, L., Hughes, R., Mitev, K., Whitmarsh, L., Demski, C., Nash, N., Papakonstantinou, T., and Wilson, M. (2025). The use of large language models for qualitative research: The Deep Computational Text Analyser (DECOTA). Psychol Methods, 1\u201320.","DOI":"10.1037\/met0000753"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"10587","DOI":"10.1073\/pnas.0903616106","article-title":"Universals and cultural variation in turn-taking in conversation","volume":"106","author":"Stivers","year":"2009","journal-title":"Proc. Natl. Acad. Sci. USA"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/6\/1958\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T05:10:30Z","timestamp":1774156230000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/6\/1958"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,20]]},"references-count":45,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["s26061958"],"URL":"https:\/\/doi.org\/10.3390\/s26061958","relation":{"has-preprint":[{"id-type":"doi","id":"10.36227\/techrxiv.177223092.27607998\/v1","asserted-by":"object"}]},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,20]]}}}