{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T15:29:26Z","timestamp":1787671766151,"version":"build-2736575974"},"reference-count":67,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T00:00:00Z","timestamp":1699833600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotica"],"published-print":{"date-parts":[[2024,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The language-guided visual robotic grasping task focuses on enabling robots to grasp objects based on human language instructions. However, real-world human-robot collaboration tasks often involve situations with ambiguous language instructions and complex scenarios. These challenges arise in the understanding of linguistic queries, discrimination of key concepts in visual and language information, and generation of executable grasping configurations for the robot\u2019s end-effector. To overcome these challenges, we propose a novel multi-modal transformer-based framework in this study, which assists robots in localizing spatial interactions of objects using text queries and visual sensing. This framework facilitates object grasping in accordance with human instructions. Our developed framework consists of two main components. First, a visual-linguistic transformer encoder is employed to model multi-modal interactions for objects referred to in the text. Second, the framework performs joint spatial localization and grasping. Extensive ablation studies have been conducted on multiple datasets to evaluate the advantages of each component in our model. Additionally, physical experiments have been performed with natural language-driven human-robot interactions on a physical robot to validate the practicality of our approach.<\/jats:p>","DOI":"10.1017\/s0263574723001510","type":"journal-article","created":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T01:01:26Z","timestamp":1699837286000},"page":"415-434","source":"Crossref","is-referenced-by-count":19,"title":["Multi-modal interaction with transformers: bridging robots and human with natural language"],"prefix":"10.1017","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6985-240X","authenticated-orcid":false,"given":"Shaochen","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhangli","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhijun","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhen","family":"Kan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2023,11,13]]},"reference":[{"key":"S0263574723001510_ref58","first-page":"4693","volume-title":"Proc. IEEE Int. Conf. Comput. Vis.","author":"Sadhu","year":"2019"},{"key":"S0263574723001510_ref44","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2911066"},{"key":"S0263574723001510_ref41","first-page":"792","volume-title":"Proc. Eur. Conf. Comput. Vis.","volume":"9908","author":"Nagaraja","year":"2016"},{"key":"S0263574723001510_ref60","first-page":"1316","volume-title":"Proc. IEEE Int. Conf. Robot. Autom.","author":"Redmon","year":"2015"},{"key":"S0263574723001510_ref12","first-page":"1","article-title":"Human-in-the-loop adaptive control of a soft exo-suit with actuator dynamics and ankle impedance adaptation","author":"Li","year":"2023","journal-title":"IEEE Trans. Cybern."},{"key":"S0263574723001510_ref36","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0965-7"},{"key":"S0263574723001510_ref7","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00220"},{"key":"S0263574723001510_ref21","first-page":"4880","volume-title":"IEEE Int. Conf. Robot. Autom.","author":"Chen","year":"2023"},{"key":"S0263574723001510_ref62","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919859066"},{"key":"S0263574723001510_ref50","first-page":"387","volume-title":"Proc. Eur. Conf. Comput. Vis.","volume":"12359","author":"Yang","year":"2020"},{"key":"S0263574723001510_ref46","first-page":"1036","volume-title":"AAAI Conf. Artif. Intell.","author":"Chen","year":"2021"},{"key":"S0263574723001510_ref57","first-page":"11645","volume-title":"AAAI Conf. Artif. Intell.","author":"Liu","year":"2020"},{"key":"S0263574723001510_ref18","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3187261"},{"key":"S0263574723001510_ref20","doi-asserted-by":"publisher","DOI":"10.1109\/ICARM54641.2022.9959267"},{"key":"S0263574723001510_ref10","first-page":"4171","volume-title":"Proc. Assoc. Comput. Linguistics","author":"Devlin","year":"2019"},{"key":"S0263574723001510_ref5","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2021.3113709"},{"key":"S0263574723001510_ref56","first-page":"1114","volume-title":"Int. Joint Conf. Artif. Intell.","author":"Yu","year":"2018"},{"key":"S0263574723001510_ref33","volume-title":"Proc. Adv. Neural Inf. Process. Syst.","volume":"30","author":"Vaswani","year":"2017"},{"key":"S0263574723001510_ref16","doi-asserted-by":"publisher","DOI":"10.1177\/0278364914549607"},{"key":"S0263574723001510_ref55","first-page":"451","volume-title":"Asian Conf. Comput. Vis.","volume":"11364","author":"Kovvuri","year":"2018"},{"key":"S0263574723001510_ref63","first-page":"7223","volume-title":"Proc. IEEE Int. Conf. Intell. Robots Syst.","author":"Zhou","year":"2018"},{"key":"S0263574723001510_ref39","first-page":"769","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst.","author":"Kumra","year":"2017"},{"key":"S0263574723001510_ref64","first-page":"4953","volume-title":"Proc. IEEE Int. Conf. Robot. Automat.","author":"Karaoguz","year":"2019"},{"key":"S0263574723001510_ref38","first-page":"4875","volume-title":"IJCAI","volume":"7","author":"Asif","year":"2018"},{"key":"S0263574723001510_ref9","volume-title":"Robotics: Science and Systems","author":"Shao","year":"2020"},{"key":"S0263574723001510_ref67","first-page":"9626","volume-title":"Proc. IEEE Int. Conf. Intell. Robots Syst.","author":"Kumra","year":"2020"},{"key":"S0263574723001510_ref31","first-page":"16888","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Huang","year":"2021"},{"key":"S0263574723001510_ref26","first-page":"4252","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Zhuang","year":"2018"},{"key":"S0263574723001510_ref51","first-page":"1","volume-title":"Proc. IEEE Conf. Multi. Exp.","author":"Du","year":"2022"},{"key":"S0263574723001510_ref4","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574722001679"},{"key":"S0263574723001510_ref23","first-page":"1960","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Wang","year":"2019"},{"key":"S0263574723001510_ref8","first-page":"4576","volume-title":"Proc. IEEE Int. Conf. Robot. Automat.","author":"Chen","year":"2021"},{"key":"S0263574723001510_ref30","first-page":"abs\/1812.03426","article-title":"Real-time referring expression comprehension by single-stage grounding network","author":"Chen","year":"2018","journal-title":"CoRR"},{"key":"S0263574723001510_ref48","first-page":"1950","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Liu","year":"2019"},{"key":"S0263574723001510_ref61","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2016.2638453"},{"key":"S0263574723001510_ref25","first-page":"4158","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Zhang","year":"2018"},{"key":"S0263574723001510_ref29","first-page":"1307","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Yu","year":"2018"},{"key":"S0263574723001510_ref24","first-page":"258","volume-title":"Proc. Eur. Conf. Comput. Vis.","volume":"11216","author":"Plummer","year":"2018"},{"key":"S0263574723001510_ref11","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920917755"},{"key":"S0263574723001510_ref53","first-page":"1749","volume-title":"Proc. IEEE Int. Conf. Comput. Vis.","author":"Deng","year":"2021"},{"key":"S0263574723001510_ref19","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2023.3309049"},{"key":"S0263574723001510_ref32","first-page":"4682","volume-title":"Proc. IEEE Int. Conf. Comput. Vis.","author":"Yang","year":"2019"},{"key":"S0263574723001510_ref6","volume-title":"See What I\u2019m Saying: The Extraordinary Powers of Our Five Senses","author":"Rosenblum","year":"2011"},{"key":"S0263574723001510_ref65","first-page":"3996","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst.","author":"Gari\u00e9py","year":"2019"},{"key":"S0263574723001510_ref37","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00166"},{"key":"S0263574723001510_ref59","first-page":"3304","volume-title":"Proc. IEEE Int. Conf. Robot. Automat.","author":"Jiang","year":"2011"},{"key":"S0263574723001510_ref66","first-page":"4768","volume-title":"Proc. IEEE Int. Conf. Intell. Robots Syst.","author":"Zhang","year":"2019"},{"key":"S0263574723001510_ref15","first-page":"13452","volume-title":"Proc. IEEE Int. Conf. Robot. Automat.","author":"Ainetter","year":"2021"},{"key":"S0263574723001510_ref22","article-title":"What you see is what you grasp: User-friendly grasping guided by near-eye-tracking","author":"Wang","year":"2022","journal-title":"arXiv preprint arXiv:2209.06122"},{"key":"S0263574723001510_ref14","doi-asserted-by":"crossref","first-page":"1687814016668077","DOI":"10.1177\/1687814016668077","article-title":"Robot grasp detection using multimodal deep convolutional neural networks","volume":"8","author":"Wang","year":"2016","journal-title":"Adv. Mech. Eng."},{"key":"S0263574723001510_ref49","first-page":"10877","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Liao","year":"2020"},{"key":"S0263574723001510_ref3","doi-asserted-by":"publisher","DOI":"10.1017\/S026357472200159X"},{"key":"S0263574723001510_ref28","first-page":"69","volume-title":"Proc. Eur. Conf. Comput. Vis.","volume":"9906","author":"Yu","year":"2016"},{"key":"S0263574723001510_ref17","first-page":"1609","volume-title":"Proc. IEEE Int. Conf. Robot. Automat.","author":"Di","year":"2017"},{"key":"S0263574723001510_ref54","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2797921"},{"key":"S0263574723001510_ref27","first-page":"11","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Mao","year":"2016"},{"key":"S0263574723001510_ref45","first-page":"4672","volume-title":"Proc. IEEE Int. Conf. Comput. Vis.","author":"Liu","year":"2019"},{"key":"S0263574723001510_ref47","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.2973983"},{"key":"S0263574723001510_ref2","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574721000722"},{"key":"S0263574723001510_ref52","first-page":"19652","volume-title":"Proc. Int. Conf. Neural Inf. Process. Syst.","author":"Li","year":"2021"},{"key":"S0263574723001510_ref43","first-page":"4643","volume-title":"Proc. IEEE Int. Conf. Comput. Vis.","author":"Yang","year":"2019"},{"key":"S0263574723001510_ref35","first-page":"3511","volume-title":"Proc. IEEE\/RSJ Int. Conf. Intell. Robots Syst.","author":"Depierre","year":"2018"},{"key":"S0263574723001510_ref1","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574720001101"},{"key":"S0263574723001510_ref13","doi-asserted-by":"publisher","DOI":"10.1109\/TMECH.2020.3044289"},{"key":"S0263574723001510_ref40","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919897133"},{"key":"S0263574723001510_ref42","first-page":"4418","volume-title":"Proc. IEEE Conf. Comput. Vis. Pattern Recognit.","author":"Hu","year":"2017"},{"key":"S0263574723001510_ref34","first-page":"213","volume-title":"Proc. Eur. Conf. Comput. Vis.","author":"Carion","year":"2020"}],"container-title":["Robotica"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0263574723001510","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T21:43:48Z","timestamp":1730497428000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0263574723001510\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,13]]},"references-count":67,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2]]}},"alternative-id":["S0263574723001510"],"URL":"https:\/\/doi.org\/10.1017\/s0263574723001510","relation":{},"ISSN":["0263-5747","1469-8668"],"issn-type":[{"value":"0263-5747","type":"print"},{"value":"1469-8668","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,13]]}}}