{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T21:18:09Z","timestamp":1774041489017,"version":"3.50.1"},"reference-count":71,"publisher":"SAGE Publications","issue":"1-3","license":[{"start":{"date-parts":[[2015,11,12]],"date-time":"2015-11-12T00:00:00Z","timestamp":1447286400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2016,1]]},"abstract":"<jats:p> It is important for a robot to be able to interpret natural language commands given by a human. In this paper, we consider performing a sequence of mobile manipulation tasks with instructions described in natural language. Given a new environment, even a simple task such as boiling water would be performed quite differently depending on the presence, location and state of the objects. We start by collecting a dataset of task descriptions in free-form natural language and the corresponding grounded task-logs of the tasks performed in an online robot simulator. We then build a library of verb\u2013environment instructions that represents the possible instructions for each verb in that environment, these may or may not be valid for a different environment and task context. We present a model that takes into account the variations in natural language and ambiguities in grounding them to robotic instructions with appropriate environment context and task constraints. Our model also handles incomplete or noisy natural language instructions. It is based on an energy function that encodes such properties in a form isomorphic to a conditional random field. We evaluate our model on tasks given in a robotic simulator and show that it successfully outperforms the state of the art with 61.8% accuracy. We also demonstrate a grounded robotic instruction sequence on a PR2 robot using the Learning from Demonstration approach. <\/jats:p>","DOI":"10.1177\/0278364915602060","type":"journal-article","created":{"date-parts":[[2015,11,14]],"date-time":"2015-11-14T01:09:05Z","timestamp":1447463345000},"page":"281-300","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":117,"title":["Tell me Dave: Context-sensitive grounding of natural language to manipulation           instructions"],"prefix":"10.1177","volume":"35","author":[{"given":"Dipendra K.","family":"Misra","sequence":"first","affiliation":[{"name":"Computer Science Department, Cornell University, USA"}]},{"given":"Jaeyong","family":"Sung","sequence":"additional","affiliation":[{"name":"Computer Science Department, Cornell University, USA"}]},{"given":"Kevin","family":"Lee","sequence":"additional","affiliation":[{"name":"Computer Science Department, Cornell University, USA"}]},{"given":"Ashutosh","family":"Saxena","sequence":"additional","affiliation":[{"name":"Computer Science Department, Cornell University, USA"}]}],"member":"179","published-online":{"date-parts":[[2015,11,12]]},"reference":[{"key":"bibr1-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-012-0160-0"},{"key":"bibr2-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5980286"},{"key":"bibr3-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1177\/0278364912461538"},{"key":"bibr4-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"bibr5-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00209"},{"key":"bibr6-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5980495"},{"key":"bibr7-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-00065-7_36"},{"key":"bibr8-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/Humanoids.2011.6100855"},{"key":"bibr9-0278364915602060","volume-title":"Proceedings of EMNLP","author":"Berant J","year":"2013"},{"key":"bibr10-0278364915602060","volume-title":"The PR2 Workshop, IROS","author":"Bollini M","year":"2011"},{"key":"bibr11-0278364915602060","volume-title":"ISER","author":"Bollini M","year":"2012"},{"key":"bibr12-0278364915602060","first-page":"1268","volume-title":"ACL\u201910 proceedings of the 48th annual meeting of the Association for Computational Linguistics","author":"Branavan S","year":"2010"},{"key":"bibr13-0278364915602060","volume-title":"EpiRob","author":"Cakmak M","year":"2007"},{"key":"bibr14-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/DEVLRN.2011.6037321"},{"key":"bibr15-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1613\/jair.2962"},{"key":"bibr16-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2013.6631000"},{"key":"bibr17-0278364915602060","volume-title":"Proceedings of the international symposium on experimental robotics (ISER)","author":"Duvallet F","year":"2014"},{"key":"bibr18-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2013.6696861"},{"key":"bibr19-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2010.5539924"},{"key":"bibr20-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15561-1_2"},{"key":"bibr21-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2013.6696345"},{"issue":"3","key":"bibr22-0278364915602060","first-page":"189","volume":"2","author":"Fikes RE","year":"1972","journal-title":"Artificial Intelligence"},{"key":"bibr23-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2010.5650371"},{"key":"bibr24-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2013.6696569"},{"key":"bibr25-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2014.X.041"},{"key":"bibr26-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2014.6907491"},{"key":"bibr27-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2010.5649494"},{"key":"bibr28-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.385"},{"key":"bibr29-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1177\/0278364912438781"},{"key":"bibr30-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5980391"},{"key":"bibr31-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2010.08.002"},{"key":"bibr32-0278364915602060","doi-asserted-by":"publisher","DOI":"10.3115\/1075096.1075150"},{"key":"bibr33-0278364915602060","volume-title":"ISER","author":"Kollar T","year":"2010"},{"key":"bibr34-0278364915602060","volume-title":"NIPS","author":"Koppula H","year":"2011"},{"key":"bibr35-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2013.IX.006"},{"key":"bibr36-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913478446"},{"key":"bibr37-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2007.4398998"},{"key":"bibr38-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2010.06.001"},{"key":"bibr39-0278364915602060","first-page":"1451","volume-title":"IJCAI\u201913 Proceedings of the twenty-third international joint conference on artificial intelligence","author":"Kulick J","year":"2013"},{"key":"bibr40-0278364915602060","first-page":"1223","volume-title":"EMNLP","author":"Kwiatkowski T","year":"2010"},{"key":"bibr41-0278364915602060","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1135"},{"issue":"2","key":"bibr42-0278364915602060","first-page":"181","volume":"4","author":"Lemaignan S","year":"2012","journal-title":"International Journal of Science and Research"},{"key":"bibr43-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2013.IX.012"},{"key":"bibr44-0278364915602060","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-2407"},{"key":"bibr45-0278364915602060","volume-title":"IAS","author":"Marco D","year":"2012"},{"key":"bibr46-0278364915602060","first-page":"1671","volume-title":"ICML","author":"Matuszek C","year":"2012"},{"key":"bibr47-0278364915602060","volume-title":"ISER","author":"Matuszek C","year":"2012"},{"key":"bibr48-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2014.X.005"},{"key":"bibr49-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2007.914848"},{"key":"bibr50-0278364915602060","volume-title":"Proceedings of the twenty eighth conference on uncertainty in artificial intelligence","author":"Mourao K","year":"2012"},{"key":"bibr51-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2013.6630616"},{"key":"bibr52-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2013.IX.048"},{"key":"bibr53-0278364915602060","volume-title":"ACL","author":"Nothman J","year":"2012"},{"key":"bibr54-0278364915602060","volume-title":"ACL","author":"Poon H","year":"2013"},{"key":"bibr55-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.117"},{"key":"bibr56-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2009.5152817"},{"key":"bibr57-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2012.08.001"},{"key":"bibr58-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2010.5598719"},{"key":"bibr59-0278364915602060","unstructured":"Saxena A, Jain A, Sener O, (2014) Robobrain: Large-scale knowledge             engine for robots. Preprint           arXiv:1412.0691."},{"key":"bibr60-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1007\/s10514-009-9160-9"},{"key":"bibr61-0278364915602060","volume-title":"Surface structure and interpretation","author":"Steedman M","year":"1996"},{"key":"bibr62-0278364915602060","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/6591.001.0001","volume-title":"The syntactic process","volume":"35","author":"Steedman M","year":"2000"},{"key":"bibr63-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2014.6942972"},{"key":"bibr64-0278364915602060","volume-title":"Collaborative manipulation workshop at human\u2013Robot interaction","author":"Tellex S","year":"2013"},{"key":"bibr65-0278364915602060","volume-title":"AAAI","author":"Tellex S","year":"2011"},{"key":"bibr66-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1109\/ICHR.2010.5686350"},{"key":"bibr67-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2013.IX.004"},{"key":"bibr68-0278364915602060","doi-asserted-by":"publisher","DOI":"10.1145\/1755952.1755968"},{"key":"bibr69-0278364915602060","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2014.X.006"},{"key":"bibr70-0278364915602060","volume-title":"ACL","author":"Yu H","year":"2013"},{"key":"bibr71-0278364915602060","volume-title":"EMNLP-CoNLL-2007","author":"Zettlemoyer L","year":"2007"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364915602060","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0278364915602060","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364915602060","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T15:06:06Z","timestamp":1740755166000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0278364915602060"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,11,12]]},"references-count":71,"journal-issue":{"issue":"1-3","published-print":{"date-parts":[[2016,1]]}},"alternative-id":["10.1177\/0278364915602060"],"URL":"https:\/\/doi.org\/10.1177\/0278364915602060","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,11,12]]}}}