{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T05:12:06Z","timestamp":1776143526196,"version":"3.50.1"},"reference-count":59,"publisher":"SAGE Publications","issue":"4-5","license":[{"start":{"date-parts":[[2017,6,12]],"date-time":"2017-06-12T00:00:00Z","timestamp":1497225600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2018,4]]},"abstract":"<jats:p> We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images independent of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. We describe two large-scale experiments that we conducted on two separate robotic platforms. In the first experiment, about 800,000 grasp attempts were collected over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and gripper wear and tear. In the second experiment, we used a different robotic platform and 8 robots to collect a dataset consisting of over 900,000 grasp attempts. The second robotic platform was used to test transfer between robots, and the degree to which data from a different set of robots can be used to aid learning. Our experimental results demonstrate that our approach achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing. Our transfer experiment also illustrates that data from different robots can be combined to learn more reliable and effective grasping. <\/jats:p>","DOI":"10.1177\/0278364917710318","type":"journal-article","created":{"date-parts":[[2017,6,12]],"date-time":"2017-06-12T09:04:16Z","timestamp":1497258256000},"page":"421-436","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1233,"title":["Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection"],"prefix":"10.1177","volume":"37","author":[{"given":"Sergey","family":"Levine","sequence":"first","affiliation":[{"name":"Google"}]},{"given":"Peter","family":"Pastor","sequence":"additional","affiliation":[{"name":"X"}]},{"given":"Alex","family":"Krizhevsky","sequence":"additional","affiliation":[{"name":"Google"}]},{"given":"Julian","family":"Ibarz","sequence":"additional","affiliation":[{"name":"Google"}]},{"given":"Deirdre","family":"Quillen","sequence":"additional","affiliation":[{"name":"Google"}]}],"member":"179","published-online":{"date-parts":[[2017,6,12]]},"reference":[{"key":"bibr1-0278364917710318","author":"Antos A","year":"2008","journal-title":"Advances in neural information processing systems"},{"key":"bibr2-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2016.7759446"},{"key":"bibr3-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386072"},{"key":"bibr4-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2013.2289018"},{"key":"bibr5-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-013-9342-3"},{"key":"bibr6-0278364917710318","author":"Chen L","year":"2014","journal-title":"arXiv preprint arXiv:1412.7062"},{"key":"bibr7-0278364917710318","doi-asserted-by":"publisher","DOI":"10.2478\/s13230-011-0012-x"},{"key":"bibr8-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/70.143350"},{"key":"bibr9-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.81"},{"key":"bibr10-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2009.5152709"},{"key":"bibr11-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2009.5354078"},{"key":"bibr12-0278364917710318","author":"Gualtieri M","year":"2016","journal-title":"arXiv preprint arXiv:1603.01564"},{"key":"bibr13-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2012.6225084"},{"key":"bibr14-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-013-9366-8"},{"key":"bibr15-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.1994.407392"},{"key":"bibr16-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2012.6225101"},{"key":"bibr17-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1177\/02783640022067878"},{"key":"bibr18-0278364917710318","volume-title":"International conference on machine learning","author":"Ioffe S","year":"2015"},{"key":"bibr19-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.1997.606723"},{"key":"bibr20-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2013.12.011"},{"key":"bibr21-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1142\/S021984361250017X"},{"key":"bibr22-0278364917710318","author":"Johannson R","year":"2007","journal-title":"Handbook of the Senses"},{"key":"bibr23-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2016.7759657"},{"key":"bibr24-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139793"},{"key":"bibr25-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2013.6631180"},{"key":"bibr26-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2014.2376492"},{"key":"bibr27-0278364917710318","volume-title":"IEEE international conference on robotics and automation","author":"Koo S","year":"2016"},{"key":"bibr28-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1177\/0278364915594244"},{"key":"bibr29-0278364917710318","first-page":"15","author":"Kragic D","year":"2002","journal-title":"Computational Vision and Active Perception Laboratory"},{"key":"bibr30-0278364917710318","author":"Krizhevsky A","year":"2012","journal-title":"Advances in neural information processing systems"},{"key":"bibr31-0278364917710318","volume-title":"IEEE-RAS international conference on humanoid robotics","author":"Kuffner J","year":"2010"},{"key":"bibr32-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2013.6707053"},{"issue":"10","key":"bibr33-0278364917710318","volume":"3361","author":"LeCun Y","year":"1995","journal-title":"The Handbook of Brain Theory and Neural Networks"},{"key":"bibr34-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28572-1_18"},{"key":"bibr35-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1177\/0278364914549607"},{"key":"bibr36-0278364917710318","author":"Levine S","year":"2015","journal-title":"arXiv preprint arXiv: 1504.00702"},{"key":"bibr37-0278364917710318","volume-title":"International symposium on experimental robotics","author":"Levine S","year":"2016"},{"key":"bibr38-0278364917710318","volume-title":"International conference on learning representations","author":"Lillicrap T","year":"2016"},{"key":"bibr39-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2016.7487342"},{"key":"bibr40-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"bibr41-0278364917710318","volume-title":"IEEE international conference on robotics and automation","author":"Mohta K","year":"2014"},{"key":"bibr42-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/DEVLRN.2009.5175529"},{"key":"bibr43-0278364917710318","volume-title":"International symposium on robotics research","author":"Oberlin J","year":"2015"},{"key":"bibr44-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1007\/s10846-016-0376-6"},{"key":"bibr45-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2016.7487517"},{"key":"bibr46-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139361"},{"key":"bibr47-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1177\/0278364912442972"},{"key":"bibr48-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-4321-0"},{"key":"bibr49-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1007\/s00221-003-1525-2"},{"key":"bibr50-0278364917710318","volume-title":"Springer Handbook of Robotics","author":"Siciliano B","year":"2007"},{"key":"bibr51-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2012.2200561"},{"key":"bibr52-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICHR.2008.4755985"},{"key":"bibr53-0278364917710318","volume-title":"Generating Multi-Fingered Robotic Grasps via Deep Learning IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Varley J","year":"2015"},{"key":"bibr54-0278364917710318","author":"Watter M","year":"2015","journal-title":"Advances in neural information processing systems"},{"key":"bibr55-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2012.6224697"},{"key":"bibr56-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2016.7487185"},{"key":"bibr57-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/70.538974"},{"key":"bibr58-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298930"},{"key":"bibr59-0278364917710318","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.1994.350995"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364917710318","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0278364917710318","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364917710318","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T07:58:57Z","timestamp":1741075137000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0278364917710318"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,12]]},"references-count":59,"journal-issue":{"issue":"4-5","published-print":{"date-parts":[[2018,4]]}},"alternative-id":["10.1177\/0278364917710318"],"URL":"https:\/\/doi.org\/10.1177\/0278364917710318","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,6,12]]}}}