{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T05:08:57Z","timestamp":1779340137817,"version":"3.51.4"},"reference-count":28,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T00:00:00Z","timestamp":1779148800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2024YFB4709400"],"award-info":[{"award-number":["2024YFB4709400"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the growing demand for assistive robots in aging societies, task-oriented grasping in household environments has become increasingly important. Compared with structured industrial settings, household scenarios are characterized by diverse objects, unstructured layouts, and strong variability in task semantics. However, traditional methods focus on geometric stability and fail to capture task-relevant semantic constraints on manipulation regions, while existing approaches suffer from unstable reasoning and lack effective mechanisms for incorporating human intervention into the reasoning process. To address these challenges, we propose Hi-RAGrasp, a task-oriented grasping framework that integrates progressive multi-stage reasoning, Human-in-the-Loop (HITL) interaction, and Retrieval-Augmented Generation (RAG). A coarse-to-fine pipeline progressively refines predictions from object-level localization to part-level grounding, enabling robust mapping from human instructions to fine-grained task-relevant regions. Meanwhile, a HITL correction mechanism and a structured human experience database are introduced and combined with RAG to form a unified paradigm that aligns with prior experience when available and falls back to reasoning otherwise, enabling experience reuse and future experience accumulation without retraining. In addition, a Geometric Heuristic Segmentation (GHS) method is proposed to improve task-relevant region localization for textureless objects. Experiments show that our method achieves a segmentation success rate of 77.73% on the evaluation dataset and a grasp success rate of 75% in real-world scenarios, significantly outperforming existing methods and demonstrating strong effectiveness and practicality in open environments.<\/jats:p>","DOI":"10.3390\/s26103221","type":"journal-article","created":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T15:17:23Z","timestamp":1779203843000},"page":"3221","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Hi-RAGrasp: A Human-in-the-Loop Experience-Augmented Method for Task-Oriented Grasping"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3694-9800","authenticated-orcid":false,"given":"Yaxin","family":"Liu","sequence":"first","affiliation":[{"name":"State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yue","family":"Hu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8655-5335","authenticated-orcid":false,"given":"Yan","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1612-0459","authenticated-orcid":false,"given":"Ming","family":"Zhong","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,5,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Gieseking, J.J., Mangold, W., Katz, C., Low, S., and Saegert, S. (2014). The Theory of Affordances:(1979). The People, Place, and Space Reader, Routledge.","DOI":"10.4324\/9781315816852"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"0120","DOI":"10.34133\/cbsystems.0120","article-title":"Fast and Comfortable Robot-to-Human Handover for Mobile Cooperation Robot System","volume":"5","author":"Meng","year":"2024","journal-title":"Cyborg Bionic Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"0572","DOI":"10.34133\/cbsystems.0572","article-title":"Dynamic manipulation skill learning for tactile myoelectric prosthetic hands in tool handling","volume":"7","author":"Li","year":"2026","journal-title":"Cyborg Bionic Syst."},{"key":"ref_4","unstructured":"Murali, A., Liu, W., Marino, K., Chernova, S., and Gupta, A. (2020, January 16\u201318). Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping. Proceedings of the 4th Conference on Robot Learning (CoRL 2020), Cambridge, MA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Song, D., Huebner, K., Kyrki, V., and Kragic, D. (2010, January 18\u201322). Learning Task Constraints for Robot Grasping Using Graphical Models. Proceedings of the 2010 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS 2010), Taipei, Taiwan.","DOI":"10.1109\/IROS.2010.5649406"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Dong, W., Huang, D., Liu, J., Tang, C., and Zhang, H. (2025, January 19\u201323). RTAGrasp: Learning Task-Oriented Grasping from Human Videos via Retrieval, Transfer, and Alignment. Proceedings of the 2025 IEEE International Conference on Robotics and Automation (ICRA 2025), Atlanta, GA, USA.","DOI":"10.1109\/ICRA55743.2025.11128140"},{"key":"ref_7","unstructured":"Kuang, Y., Ye, J., Geng, H., Mao, J., Deng, C., Guibas, L., Wang, H., and Wang, Y. (2026, March 20). RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation. Available online: https:\/\/arxiv.org\/pdf\/2407.04689."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Luo, H., Zhai, W., Zhang, J., Cao, Y., and Tao, D. (2022, January 18\u201324). Learning Affordance Grounding from Exocentric Images. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00229"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Gao, X., Zhang, P., Qu, D., Wang, D., Wang, Z., Ding, Y., and Zhao, B. (2026, March 20). Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding. Available online: https:\/\/arxiv.org\/pdf\/2408.13024.","DOI":"10.1609\/aaai.v39i3.32318"},{"key":"ref_10","unstructured":"Jang, J.H., Seo, H., and Chun, S.Y. (2026, March 20). INTRA: Interaction Relationship-Aware Weakly Supervised Affordance Grounding 2024. Available online: https:\/\/arxiv.org\/pdf\/2409.06210."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Wang, Y., Wu, A., Yang, M., Min, Y., Zhu, Y., and Deng, C. (2025, January 11\u201315). Reasoning Mamba: Hypergraph-Guided Region Relation Calculating for Weakly Supervised Affordance Grounding. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025), Nashville, TN, USA.","DOI":"10.1109\/CVPR52734.2025.02572"},{"key":"ref_12","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2026, March 20). Learning Transferable Visual Models from Natural Language Supervision. Available online: https:\/\/arxiv.org\/pdf\/2103.00020."},{"key":"ref_13","unstructured":"Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., and El-Nouby, A. (2026, March 20). DINOv2: Learning Robust Visual Features without Supervision. Available online: https:\/\/arxiv.org\/pdf\/2304.07193."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Cuttano, C., Rosi, G., Trivigno, G., and Averta, G. (2024, January 17\u201321). What Does CLIP Know about Peeling a Banana?. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2024), Seattle, WA, USA.","DOI":"10.1109\/CVPRW63382.2024.00229"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"19589","DOI":"10.1109\/TNNLS.2025.3591538","article-title":"Learning Granularity-Aware Affordances from Human-Object Interaction for Tool-Based Functional Dexterous Grasping","volume":"36","author":"Yang","year":"2025","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_16","unstructured":"Li, D., Zhao, C., Yang, S., Ma, L., Li, Y., and Zhang, W. (2024, January 3\u20135). Learning Instruction-Guided Manipulation Affordance. Proceedings of the International Conference on Advanced Robotics and Mechatronics (ICARM 2024), Tokyo, Japan."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Li, G., Sun, D., Sevilla-Lara, L., and Jampani, V. (2024, January 17\u201321). One-Shot Open Affordance Learning with Foundation Models. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.00298"},{"key":"ref_18","unstructured":"Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., and Fei-Fei, L. (2026, March 20). VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models 2023. Available online: https:\/\/arxiv.org\/pdf\/2307.05973."},{"key":"ref_19","unstructured":"Huang, H., Lin, F., Hu, Y., Wang, S., and Gao, Y. (2024, January 14\u201318). Copa. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS 2024), Abu Dhabi, United Arab Emirates."},{"key":"ref_20","unstructured":"Tang, Y., Zhang, S., Hao, X., Wang, P., Wu, J., Wang, Z., and Zhang, S. (2025, January 19\u201325). AffordGrasp. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS 2025), Hangzhou, China."},{"key":"ref_21","unstructured":"Huang, W., Wang, C., Li, Y., Zhang, R., and Fei-Fei, L. (2026, March 20). ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. Available online: https:\/\/arxiv.org\/pdf\/2409.01652."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Korekata, R., Xie, Q., Bisk, Y., and Sugiura, K. (2026, March 20). Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation. Available online: https:\/\/arxiv.org\/pdf\/2512.18987.","DOI":"10.1109\/LRA.2026.3653281"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Chen, M., Bai, Y., Fang, X., and Yi, P. (2025, January 20\u201322). Adaptive Retrieval-Augmented Embodied Intelligence RAG Reasoning Framework. Proceedings of the 2025 IEEE 5th International Conference on Software Engineering and Artificial Intelligence (SEAI 2025), Fuzhou, China.","DOI":"10.1109\/SEAI65851.2025.11108876"},{"key":"ref_24","unstructured":"Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., and Yan, F. (2026, March 20). Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. Available online: https:\/\/arxiv.org\/pdf\/2401.14159."},{"key":"ref_25","unstructured":"Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. (2026, March 20). Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. Available online: https:\/\/arxiv.org\/pdf\/2310.11441."},{"key":"ref_26","unstructured":"Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Yang, J., Li, C., Zhang, L., and Gao, J. (2026, March 20). Semantic-SAM: Segment and Recognize Anything at Any Granularity. Available online: https:\/\/arxiv.org\/pdf\/2307.04767."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3929","DOI":"10.1109\/TRO.2023.3281153","article-title":"AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains","volume":"39","author":"Fang","year":"2023","journal-title":"IEEE Trans. Robot."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Li, G., Tsagkas, N., Song, J., Mon-Williams, R., Vijayakumar, S., Shao, K., and Sevilla-Lara, L. (2025, January 11\u201315). Learning Precise Affordances from Egocentric Videos for Robotic Manipulation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025), Nashville, TN, USA.","DOI":"10.1109\/ICCV51701.2025.00985"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/10\/3221\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T04:29:53Z","timestamp":1779337793000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/10\/3221"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,19]]},"references-count":28,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,5]]}},"alternative-id":["s26103221"],"URL":"https:\/\/doi.org\/10.3390\/s26103221","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,19]]}}}