{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:04:43Z","timestamp":1753891483246,"version":"3.41.2"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,1,10]],"date-time":"2023-01-10T00:00:00Z","timestamp":1673308800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>As robots begin to collaborate with humans in their daily work spaces, they need to have a deeper understanding of the tasks of using tools. In response to the problem of using tools in collaboration between humans and robots, we propose a modular system based on collaborative tasks. The first part of the system is designed to find task-related operating areas, and a multi-layer instance segmentation network is used to find the tools needed for the task, and classify the object itself based on the state of the robot in the collaborative task. Thus, we generate the state semantic region with the \u201cleader-assistant\u201d state. In the second part, in order to predict the optimal grasp and handover configuration, a multi-scale grasping network (MGR-Net) based on the mask of state semantic area is proposed, it can better adapt to the change of the receptive field caused by the state semantic region. Compared with the traditional method, our method has higher accuracy. The whole system also achieves good results on untrained real-world tool dataset we constructed. To further verify the effectiveness of our generated grasp representations, A robot platform based on Sawyer is used to prove the high performance of our system.<\/jats:p>","DOI":"10.3389\/fnbot.2022.1082550","type":"journal-article","created":{"date-parts":[[2023,1,10]],"date-time":"2023-01-10T21:21:34Z","timestamp":1673385694000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["A multi-scale robotic tool grasping method for robot state segmentation masks"],"prefix":"10.3389","volume":"16","author":[{"given":"Tao","family":"Xue","sequence":"first","affiliation":[]},{"given":"Deshuai","family":"Zheng","sequence":"additional","affiliation":[]},{"given":"Jin","family":"Yan","sequence":"additional","affiliation":[]},{"given":"Yong","family":"Liu","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,1,10]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1393","DOI":"10.1007\/s10514-018-9784-8","article-title":"Semantic and geometric reasoning for robotic grasping: a probabilistic logic approach","volume":"43","author":"Antanas","year":"2019","journal-title":"Auton. Robots"},{"key":"B2","doi-asserted-by":"publisher","first-page":"8709","DOI":"10.1109\/CVPR.2019.00891","article-title":"\u201cContactDB: analyzing and predicting grasp contact via thermal imaging,\u201d","author":"Brahmbhatt","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B3","doi-asserted-by":"publisher","first-page":"8573","DOI":"10.1109\/CVPR42600.2020.00860","article-title":"\u201cBlendmask: top-down meets bottom-up for instance segmentation,\u201d","author":"Chen","year":"2020","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B4","doi-asserted-by":"publisher","first-page":"834","DOI":"10.1109\/TPAMI.2017.2699184","article-title":"Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs","volume":"40","author":"Chen","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1109\/LRA.2019.2894439","article-title":"Learning affordance segmentation for real-world robotic manipulation via synthetic images","volume":"4","author":"Chu","year":"2019","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B6","doi-asserted-by":"publisher","first-page":"3511","DOI":"10.1109\/IROS.2018.8593950","article-title":"\u201cJacquard: a large scale dataset for robotic grasp detection,\u201d","author":"Depierre","year":"2018","journal-title":"2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B7","doi-asserted-by":"publisher","first-page":"4370","DOI":"10.1109\/ICRA48506.2021.9561198","article-title":"\u201cScoring graspability based on grasp regression for better grasp prediction,\u201d","author":"Depierre","year":"2021","journal-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B8","doi-asserted-by":"publisher","first-page":"5882","DOI":"10.1109\/ICRA.2018.8460902","article-title":"\u201cAffordanceNet: an end-to-end deep learning approach for object affordance detection,\u201d","author":"Do","year":"2018","journal-title":"2018 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B9","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1007\/978-3-031-13841-6_40","article-title":"\u201cRobotic grasp detection based on transformer,\u201d","volume-title":"International Conference on Intelligent Robotics and Applications","author":"Dong","year":"2022"},{"key":"B10","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1016\/j.comcom.2021.07.012","article-title":"Mask-GD segmentation based robotic grasp detection","volume":"178","author":"Dong","year":"2021","journal-title":"Comput. Commun"},{"key":"B11","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1177\/0278364919872545","article-title":"Learning task-oriented grasping for tool manipulation from simulated self-supervision","volume":"39","author":"Fang","year":"2020","journal-title":"Int. J. Robot. Res"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1167","DOI":"10.1177\/0278364915577105","article-title":"Learning grasps with topographic features","volume":"34","author":"Fischinger","year":"2015","journal-title":"Int. J. Robot. Res"},{"key":"B13","doi-asserted-by":"publisher","first-page":"4019","DOI":"10.1109\/IROS.2016.7759592","article-title":"\u201cOne-shot learning of manipulation skills with online dynamics adaptation and neural network priors,\u201d","author":"Fu","year":"2016","journal-title":"2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B14","doi-asserted-by":"publisher","first-page":"642","DOI":"10.1109\/ICCV.2019.00073","article-title":"\u201cSSAP: single-shot instance segmentation with affinity pyramid,\u201d","author":"Gao","year":"2019","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989191","article-title":"\u201cA hybrid deep architecture for robotic grasp detection,\u201d","author":"Guo","year":"2017","journal-title":"2017 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7140073","article-title":"\u201cThe affordance template ROS package for robot task programming,\u201d","author":"Hart","year":"2015","journal-title":"2015 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B17","doi-asserted-by":"publisher","first-page":"2961","DOI":"10.1109\/ICCV.2017.322","article-title":"\u201cMask R-CNN,\u201d","author":"He","year":"2017","journal-title":"Proceedings of the IEEE International Conference on Computer Vision"},{"key":"B18","doi-asserted-by":"publisher","first-page":"3304","DOI":"10.1109\/ICRA.2011.5980145","article-title":"\u201cEfficient grasping from RGBD images: learning using a new rectangle representation,\u201d","author":"Jiang","year":"2011","journal-title":"2011 IEEE International Conference on Robotics and Automation"},{"key":"B19","doi-asserted-by":"publisher","first-page":"1503","DOI":"10.1109\/ICRA.2015.7139389","article-title":"\u201cTowards learning hierarchical skills for multi-phase manipulation tasks,\u201d","author":"Kroemer","year":"2015","journal-title":"2015 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B20","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1016\/j.robot.2011.05.009","article-title":"Object-action complexes: Grounded abstractions of sensory-motor processes","volume":"59","author":"Kr\u00fcger","year":"2011","journal-title":"Robot. Auton. Syst"},{"key":"B21","doi-asserted-by":"publisher","first-page":"9626","DOI":"10.1109\/IROS45743.2020.9340777","article-title":"\u201cAntipodal robotic grasping using generative residual convolutional neural network,\u201d","author":"Kumra","year":"2020","journal-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B22","doi-asserted-by":"publisher","first-page":"769","DOI":"10.1109\/IROS.2017.8202237","article-title":"\u201cRobotic grasp detection using deep convolutional neural networks,\u201d","author":"Kumra","year":"2017","journal-title":"2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B23","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1177\/0278364914549607","article-title":"Deep learning for detecting robotic grasps","volume":"34","author":"Lenz","year":"2015","journal-title":"Int. J. Robot. Res"},{"key":"B24","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1177\/0278364917710318","article-title":"Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection","volume":"37","author":"Levine","year":"2018","journal-title":"Int. J. Robot. Res"},{"key":"B25","doi-asserted-by":"publisher","first-page":"3496","DOI":"10.1109\/ICCV.2017.378","article-title":"\u201cSGN: sequential grouping networks for instance segmentation,\u201d","author":"Liu","year":"2017","journal-title":"Proceedings of the IEEE International Conference on Computer Vision"},{"key":"B26","doi-asserted-by":"publisher","first-page":"2550","DOI":"10.1109\/ICRA40945.2020.9197289","article-title":"\u201cCage: context-aware grasping engine,\u201d","author":"Liu","year":"2020","journal-title":"2020 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B27","doi-asserted-by":"publisher","first-page":"11976","DOI":"10.1109\/CVPR52688.2022.01167","article-title":"\u201cA convNet for the 2020s,\u201d","author":"Liu","year":"2022","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2017.XIII.058","article-title":"Dex-Net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics","author":"Mahler","year":"2017","journal-title":"arXiv preprint arXiv:1703.09312"},{"key":"B29","article-title":"How to be helpful? Implementing supportive behaviors for human-robot collaboration","author":"Mangin","year":"2017","journal-title":"arXiv preprint arXiv:1710.11194"},{"key":"B30","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1177\/0278364919859066","article-title":"Learning robust, real-time, reactive robotic grasping","volume":"39","author":"Morrison","year":"2020","journal-title":"Int. J. Robot. Res"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139369","article-title":"\u201cAffordance of object parts from geometric features,\u201d","author":"Myers","year":"2014","journal-title":"Workshop on Vision meets Cognition, CVPR, Vol. 9"},{"key":"B32","doi-asserted-by":"publisher","first-page":"5908","DOI":"10.1109\/IROS.2017.8206484","article-title":"\u201cObject-based affordances detection with convolutional neural networks and dense conditional random fields,\u201d","author":"Nguyen","year":"2017","journal-title":"2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B33","doi-asserted-by":"publisher","first-page":"6831","DOI":"10.1109\/ICRA.2018.8463204","article-title":"\u201cGrasping of unknown objects using deep convolutional neural networks based on depth images,\u201d","author":"Schmidt","year":"2018","journal-title":"2018 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B34","doi-asserted-by":"publisher","first-page":"9952","DOI":"10.1007\/s10489-021-03011-z","article-title":"Generative model based robotic grasp pose prediction with limited dataset","volume":"52","author":"Shukla","year":"2022","journal-title":"Appl. Intell"},{"key":"B35","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1007\/978-3-319-51532-8_19","article-title":"\u201cUsing geometry to detect grasp poses in 3D point clouds,\u201d","volume-title":"Robotics Research","author":"Ten Pas","year":"2018"},{"key":"B36","doi-asserted-by":"publisher","first-page":"9627","DOI":"10.1109\/ICCV.2019.00972","article-title":"\u201cFCOS: fully convolutional one-stage object detection,\u201d","author":"Tian","year":"2019","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B37","first-page":"649","article-title":"\u201cSolo: segmenting objects by locations,\u201d","volume-title":"European Conference on Computer Vision","author":"Wang","year":""},{"key":"B38","first-page":"17721","article-title":"Solov2: dynamic and fast instance segmentation","volume":"33","author":"Wang","year":"","journal-title":"Adv. Neural Inform. Process. Syst"},{"key":"B39","doi-asserted-by":"publisher","first-page":"3750","DOI":"10.1109\/ICRA.2018.8461044","article-title":"\u201cRobotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching,\u201d","author":"Zeng","year":"2018","journal-title":"2018 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B40","doi-asserted-by":"publisher","first-page":"14321","DOI":"10.1007\/s00521-019-04336-0","article-title":"Object affordance detection with relationship-aware network","volume":"32","author":"Zhao","year":"2020","journal-title":"Neural Comput. Appl"},{"key":"B41","doi-asserted-by":"publisher","first-page":"7223","DOI":"10.1109\/IROS.2018.8594116","article-title":"\u201cFully convolutional grasp detection network with oriented anchor box,\u201d","author":"Zhou","year":"2018","journal-title":"2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2022.1082550\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,10]],"date-time":"2023-01-10T21:21:39Z","timestamp":1673385699000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2022.1082550\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,10]]},"references-count":41,"alternative-id":["10.3389\/fnbot.2022.1082550"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2022.1082550","relation":{},"ISSN":["1662-5218"],"issn-type":[{"type":"electronic","value":"1662-5218"}],"subject":[],"published":{"date-parts":[[2023,1,10]]},"article-number":"1082550"}}