{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,4]],"date-time":"2025-07-04T05:38:30Z","timestamp":1751607510841,"version":"3.40.5"},"reference-count":40,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2022,3,25]],"date-time":"2022-03-25T00:00:00Z","timestamp":1648166400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotica"],"published-print":{"date-parts":[[2023,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\u201cPicking out the impurities\u201d is a typical scenario in production line which is both time consuming and laborious. In this article, we propose a target-oriented robotic push-grasping system which is able to actively discover and pick the impurities in dense environments with the synergies between pushing and grasping actions. First, we propose an attention module, which includes target saliency detection and density-based occluded-region inference. Without the necessity of expensive labeling of semantic segmentation, our attention module can quickly locate the targets in the view or predict the candidate regions where the targets are most likely to be occluded. Second, we propose a push\u2013grasp synergy framework to sequentially select proper actions in different situations until all targets are picked out. Moreover, we introduce an active pushing mechanism based on a novel metric, namely Target-Centric Dispersion Degree (TCDD) for better grasping. TCDD describes whether the targets are isolated from the surrounding objects. With this metric, the robot becomes more focused on the actions around the targets and push irrelevant objects away. Experimental results on both simulated environment and real-world environment show that our proposed system outperforms several baseline approaches,which also has the capability to be generalized to new scenarios.<\/jats:p>","DOI":"10.1017\/s0263574722000297","type":"journal-article","created":{"date-parts":[[2022,3,25]],"date-time":"2022-03-25T07:01:54Z","timestamp":1648191714000},"page":"470-485","source":"Crossref","is-referenced-by-count":9,"title":["Picking out the Impurities: Attention-based Push-Grasping in Dense Clutter"],"prefix":"10.1017","volume":"41","author":[{"given":"Ning","family":"Lu","sequence":"first","affiliation":[]},{"given":"Yinghao","family":"Cai","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3374-5845","authenticated-orcid":false,"given":"Tao","family":"Lu","sequence":"additional","affiliation":[]},{"given":"Xiaoge","family":"Cao","sequence":"additional","affiliation":[]},{"given":"Weiyan","family":"Guo","sequence":"additional","affiliation":[]},{"given":"Shuo","family":"Wang","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2022,3,25]]},"reference":[{"key":"S0263574722000297_ref15","doi-asserted-by":"crossref","unstructured":"[15] C. Mitash, A. Boularias and K. Bekris, \u201cRobust 6D Object Pose Estimation with Stochastic Congruent Sets,\u201d In: British Machine Vision Conference (2018).","DOI":"10.1109\/ICRA.2018.8461163"},{"key":"S0263574722000297_ref3","doi-asserted-by":"crossref","unstructured":"[3] Kiatos, M. and Malassiotis. S.,\u201cRobust object grasping in clutter via singulation,\u201d In: International Conference on Robotics and Automation (ICRA)(2019, pp. 1596\u20131600.","DOI":"10.1109\/ICRA.2019.8793972"},{"key":"S0263574722000297_ref8","unstructured":"[8] Boularias, A. , Bagnell, J. A. and Stentz, A. , \u201cLearning to Manipulate Unknown Objects in Clutter by Reinforcement,\u201d In: The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI\u201915) (AAAI Press) pp. 1336-1342."},{"key":"S0263574722000297_ref25","doi-asserted-by":"crossref","unstructured":"[25] Omr\u010den, D. , B\u00f6ge, C. , Asfour, T. , Ude, A. and Dillmann, R. , \u201cAutonomous Acquisition of Pushing Actions to Support Object Grasping with a Humanoid Robot,\u201d In: IEEE-RAS International Conference on Humanoid Robots (2009) pp. 277\u2013283.","DOI":"10.1109\/ICHR.2009.5379566"},{"key":"S0263574722000297_ref4","doi-asserted-by":"crossref","unstructured":"[4] Danielczuk, M. , Kurenkov, A. , Balakrishna, A. et al., \u201cMechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter,\u201d In: International Conference on Robotics and Automation (ICRA) (2019) pp. 1614\u20131621.","DOI":"10.1109\/ICRA.2019.8794143"},{"key":"S0263574722000297_ref18","doi-asserted-by":"crossref","unstructured":"[18] Zeng, A. , Yu, K. , Song, S. et al., \u201cMulti-view Self-Supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge,\u201d In: IEEE International Conference on Robotics and Automation (ICRA) (2017) pp. 1386\u20131383.","DOI":"10.1109\/ICRA.2017.7989165"},{"key":"S0263574722000297_ref40","doi-asserted-by":"crossref","unstructured":"[40] Bousmalis, K. , Silberman, N. D. and etal, D. , \u201cUnsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks,\u201d In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) pp. 3722\u20133731.","DOI":"10.1109\/CVPR.2017.18"},{"key":"S0263574722000297_ref1","doi-asserted-by":"crossref","unstructured":"[1] Deng, Y. , Guo, X. and Wei, Y. , et al.,\u201cDeep Reinforcement Learning for Robotic Pushing and Picking in Cluttered Environment,\u201d In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (2019), pp. 619\u2013626.","DOI":"10.1109\/IROS40897.2019.8967899"},{"key":"S0263574722000297_ref33","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/34.730558","article-title":"A model of saliency-based visual attention for rapid scene analysis","volume":"20","author":"Itti","year":"1998","journal-title":"IEEE Trans. Patt. Anal."},{"key":"S0263574722000297_ref11","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1016\/j.robot.2011.07.016","article-title":"An overview of 3D object grasp synthesis algorithms","volume":"60","author":"Sahbani","year":"2012","journal-title":"Robot. Auton. Syst."},{"key":"S0263574722000297_ref36","doi-asserted-by":"crossref","unstructured":"[36] Huang, G. , Liu, Z. , Maaten, L. V. D. and Weinberger, K. Q. , \u201cDensely Connected Convolutional Networks,\u201d In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) pp. 2261\u20132269.","DOI":"10.1109\/CVPR.2017.243"},{"key":"S0263574722000297_ref38","doi-asserted-by":"crossref","unstructured":"[38] Qin, X. , Zhang, Z. , Huang, C. , Gao, C. , Dehghan, M. and Jagersand, M. , \u201cBASNet: Boundary-Aware Salient Object Detection,\u201d In: IEEE Conference on Computer Vision and Patter Recognition (CVPR) (2019) pp. 7479\u20137489.","DOI":"10.1109\/CVPR.2019.00766"},{"key":"S0263574722000297_ref37","doi-asserted-by":"crossref","unstructured":"[37] Rohmer, E. , Singh, S. P. N. and Freese, M. , \u201cV-REP: A Versatile and Scalable Robot Simulation Framework,\u201d In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (2013) pp. 1321\u20131326.","DOI":"10.1109\/IROS.2013.6696520"},{"key":"S0263574722000297_ref31","doi-asserted-by":"crossref","first-page":"2941","DOI":"10.1109\/TCSVT.2018.2870832","article-title":"Review of visual saliency detection with comprehensive information","volume":"29","author":"Cong","year":"2019","journal-title":"IEEE Trans. Circ. Syst. Vid."},{"key":"S0263574722000297_ref29","doi-asserted-by":"crossref","unstructured":"[29] Kurenkov, A. , Taglic, J. , Kulkarni, R. et al., \u201cVisuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter,\u201d In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (2020) pp. 8408\u20138414.","DOI":"10.1109\/IROS45743.2020.9341545"},{"key":"S0263574722000297_ref20","first-page":"1","article-title":"Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching","volume":"56","author":"Zeng","year":"2019","journal-title":"Int. J. Robot. Res"},{"key":"S0263574722000297_ref19","doi-asserted-by":"crossref","unstructured":"[19] Pinto, L. and Gupta, A. , \u201cSupersizing Self-supervision: Learning to Grasp from 50K tries and 700 Robot Hours,\u201d In: IEEE International Conference on Robotics and Automation (ICRA) (2016) pp. 3406\u20133413.","DOI":"10.1109\/ICRA.2016.7487517"},{"key":"S0263574722000297_ref22","doi-asserted-by":"crossref","unstructured":"[22] Mahler, J. , Matl, M. , Liu, X. , Li, A. , Gealy, D. and Goldberg, K. , \u201cDex-Net 3.0: Computing Robust Vacuum Suction Grasp Targets in Point Clouds Using a New Analytic Model and Deep Learning,\u201d In: IEEE International Conference on Robotics and Automation (ICRA) (2018) pp. 5620\u20135627.","DOI":"10.1109\/ICRA.2018.8460887"},{"key":"S0263574722000297_ref10","doi-asserted-by":"crossref","unstructured":"[10] Liu, H. , Yuan, Y. , Deng, Y. et al., \u201cActive Affordance Exploration for Robot Grasping,\u201d In: International Conference on Intelligent Robotics and Applications (ICIRA) (2019), pp. 426\u2013438.","DOI":"10.1007\/978-3-030-27541-9_35"},{"key":"S0263574722000297_ref23","doi-asserted-by":"crossref","first-page":"3355","DOI":"10.1109\/LRA.2018.2852777","article-title":"Real-World multi-object, multi-grasp detection","volume":"3","author":"Chu","year":"2018","journal-title":"IEEE Robot. Automat. Lett"},{"key":"S0263574722000297_ref39","unstructured":"[39] Schaul, T. , Quan, J. , Antonoglou, I. and Silver, D. , \u201cPrioritized Experience Replay,\u201d In: International Conference on Learning Representations (ICLR) (2016)."},{"key":"S0263574722000297_ref32","doi-asserted-by":"crossref","unstructured":"[32] Frintrop, S. , Werner, T. and Garc\u00eda, G. M. , \u201cTraditional Saliency Reloaded: A Good Old Model in New Shape,\u201d In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015) pp. 82\u201390.","DOI":"10.1109\/CVPR.2015.7298603"},{"key":"S0263574722000297_ref28","doi-asserted-by":"crossref","unstructured":"[28] Fang, K. , Bai, Y. , Hinterstoisser, S. , Savarese, S. and Kalakrishnan, M. , \u201cMulti-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation,\u201d In: IEEE International Conference on Robotics and Automation (ICRA) (2018) pp. 3516\u20133523.","DOI":"10.1109\/ICRA.2018.8461041"},{"key":"S0263574722000297_ref35","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"S0263574722000297_ref13","doi-asserted-by":"crossref","first-page":"1849","DOI":"10.1017\/S0263574721000023","article-title":"Comprehensive review on reaching and grasping of objects in robotics","volume":"39","author":"Marwan","year":"2021","journal-title":"Robotica"},{"key":"S0263574722000297_ref34","first-page":"11","volume-title":"Link\u00f6pings Universitet\u2013ITN, Data Mining TNM033","author":"B\u00e4cklund","year":"2011"},{"key":"S0263574722000297_ref17","unstructured":"[17] Jiang, Y. , Moseson, S. and Saxena, A. , \u201cEfficient Grasping from RGBD Images: Learning Using a New Rectangle Representation,\u201d In: IEEE International Conference on Robotics and Automation(ICRA) (2011) pp. 3304\u20133311."},{"key":"S0263574722000297_ref16","unstructured":"[16] Ponce, J. , Sullivan, S. , Boissonnat, J. D. and Merlet, J.-P. , \u201cOn Characterizing and Computing Three- and Four-Finger Force-Closure Grasps of Polyhedral Objects,\u201d In: IEEE International Conference on Robotics and Automation (ICRA) (1993) pp. 821\u2013827."},{"key":"S0263574722000297_ref12","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1109\/TRO.2013.2289018","article-title":"Data-driven grasp synthesis\u2014A survey","volume":"30","author":"Bohg","year":"2014","journal-title":"IEEE Trans. Robot."},{"key":"S0263574722000297_ref21","doi-asserted-by":"crossref","first-page":"2370","DOI":"10.1109\/LRA.2018.2810544","article-title":"Learning object grasping for soft robot hands","volume":"3","author":"Choi","year":"2018","journal-title":"IEEE Robot. Automat. Lett"},{"key":"S0263574722000297_ref7","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1007\/s10514-012-9306-z","article-title":"A planning framework for Non-Prehensile manipulation under clutter and uncertainty","volume":"33","author":"Dogar","year":"2012","journal-title":"Auton. Robot."},{"key":"S0263574722000297_ref24","doi-asserted-by":"crossref","unstructured":"[24] Fang, H. , Wang, C. , Gou, M. and Lu, C. , \u201cGraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping,\u201d In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020) pp. 11444\u201311453.","DOI":"10.1109\/CVPR42600.2020.01146"},{"key":"S0263574722000297_ref30","doi-asserted-by":"crossref","first-page":"2232","DOI":"10.1109\/LRA.2020.2970622","article-title":"A deep learning approach to grasping the invisible","volume":"5","author":"Yang","year":"2020","journal-title":"IEEE Robot. Automat. Lett"},{"key":"S0263574722000297_ref6","doi-asserted-by":"crossref","unstructured":"[6] Gupta, M. and Sukhatme, G. S. , \u201cUsing Manipulation Primitives for Brick Sorting in Clutter,\u201d In: IEEE International Conference on Robotics and Automation(ICRA) (2012), pp. 3883\u20133889.","DOI":"10.1109\/ICRA.2012.6224787"},{"key":"S0263574722000297_ref27","unstructured":"[27] Jang, E. , Devin, C. , Vanhoucke, V. and Levine, S. , \u201cGrasp2Vec: Learning Object Representations from Self-Supervised Grasping,\u201d In: Proceedings of The 2nd Conference on Robot Learning, CoRL 2018, PMLR, vol. 87, (2018) pp. 99\u2013112."},{"key":"S0263574722000297_ref2","unstructured":"[2] E. Jang, S. Vijayanarasimhan, P. Pastor, J. Ibarz and S. Levine, \u201cEnd-to-End Learning of Semantic Grasping,\u201d In: Proceedings of The 1st Conference on Robot Learning, CoRL 2017, vol. 78, PMLR 2017) pp. 119\u2013132."},{"key":"S0263574722000297_ref14","unstructured":"[14] Ding, D. and Liu, Wang, S.,\u201cComputing Y. , \u201c3-D Optimal Form-Closure Grasps,\u201d In: International Conference on Robotics and Automation (ICRA) (2000), pp. 3573\u20133578."},{"key":"S0263574722000297_ref26","doi-asserted-by":"crossref","unstructured":"[26] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez and T. Funkhouser, \u201cLearning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning,\u201d In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (2018) pp. 4238\u20134245.","DOI":"10.1109\/IROS.2018.8593986"},{"key":"S0263574722000297_ref9","unstructured":"[9] Ibarz, J. Kalashnikov, D., Pastor, P. et al., \u201cScalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation,\u201d In: Proceedings of The 2nd Conference on Robot Learning, CoRL 2018, PMLR, vol. 87, (2018) pp. 651\u2013673."},{"key":"S0263574722000297_ref5","doi-asserted-by":"crossref","unstructured":"[5] Danielczuk, M. , Angelova, A. , Vanhoucke, V. and Goldberg, K. , \u201cX-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions,\u201d In: IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (2020), pp. 9577\u20139584.","DOI":"10.1109\/IROS45743.2020.9340984"}],"container-title":["Robotica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0263574722000297","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T23:52:50Z","timestamp":1726876370000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0263574722000297\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,25]]},"references-count":40,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,2]]}},"alternative-id":["S0263574722000297"],"URL":"https:\/\/doi.org\/10.1017\/s0263574722000297","relation":{},"ISSN":["0263-5747","1469-8668"],"issn-type":[{"type":"print","value":"0263-5747"},{"type":"electronic","value":"1469-8668"}],"subject":[],"published":{"date-parts":[[2022,3,25]]}}}