{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T14:35:08Z","timestamp":1761921308824,"version":"3.41.2"},"reference-count":39,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,11,6]],"date-time":"2024-11-06T00:00:00Z","timestamp":1730851200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Slip detection is crucial for achieving stable grasping and subsequent operational tasks. A grasp action is a continuous process that requires information from multiple sources. The success of a specific grasping maneuver is contingent upon the confluence of two factors: the spatial accuracy of the contact and the stability of the continuous process.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>In this paper, for the task of perceiving grasping results using visual-haptic information, we propose a new method for slip detection, which synergizes visual and haptic information from spatial-temporal dual dimensions. Specifically, the method takes as input a sequence of visual images from a first-person perspective and a sequence of haptic images from a gripper. Then, it extracts time-dependent features of the whole process and spatial features matching the importance of different parts with different attention mechanisms. Inspired by neurological studies, during the information fusion process, we adjusted temporal and spatial information from vision and haptic through a combination of two-step fusion and gate units.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results and discussion<\/jats:title><jats:p>To validate the effectiveness of method, we compared it with traditional CNN net and models with attention. It is anticipated that our method achieves a classification accuracy of 93.59%, which is higher than that of previous works. Attention visualization is further presented to support the validity.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fnbot.2024.1484751","type":"journal-article","created":{"date-parts":[[2024,11,6]],"date-time":"2024-11-06T06:12:28Z","timestamp":1730873548000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Vahagn: VisuAl Haptic Attention Gate Net for slip detection"],"prefix":"10.3389","volume":"18","author":[{"given":"Jinlin","family":"Wang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yulong","family":"Ji","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hongyu","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,11,6]]},"reference":[{"key":"B1","article-title":"\u201cThe feeling of success: does touch sensing help predict grasp outcomes,\u201d","volume-title":"Conference on Robot Learning","author":"Calandra","year":"2017"},{"key":"B2","doi-asserted-by":"publisher","first-page":"5827","DOI":"10.1109\/LRA.2020.3010720","article-title":"Self-attention based visual-tactile fusion learning for predicting grasp outcomes","volume":"5","author":"Cui","year":"2020","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B3","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1109\/IROS.2017.8202149","article-title":"\u201cImproved gelsight tactile sensor for measuring geometry and slip,\u201d","volume-title":"2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Dong","year":"2017"},{"key":"B4","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1038\/415429a","article-title":"Humans integrate visual and haptic information in a statistically optimal fashion","volume":"415","author":"Ernst","year":"2002","journal-title":"Nature"},{"key":"B5","doi-asserted-by":"publisher","first-page":"2784","DOI":"10.1109\/TASE.2022.3200073","article-title":"Tactonet: tactile ordinal network based on unimodal probability for object hardness classification","volume":"20","author":"Fang","year":"2023","journal-title":"IEEE Trans. Autom. Sci. Eng"},{"key":"B6","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1109\/TOH.2020.3004388","article-title":"Haptic and visual feedback assistance for dual-arm robot teleoperation in surface conditioning tasks","volume":"14","author":"Girbes-Juan","year":"2020","journal-title":"IEEE Trans. Haptics"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2112.06374","article-title":"Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer","author":"Han","year":"2023","journal-title":"arXiv"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"B9","doi-asserted-by":"publisher","first-page":"5985","DOI":"10.1007\/s11661-020-06008-4","article-title":"Overview: Computer vision and machine learning for microstructural characterization and analysis","volume":"51","author":"Holm","year":"2020","journal-title":"Metall. Mater. Trans. A"},{"key":"B10","doi-asserted-by":"crossref","first-page":"7132","DOI":"10.1109\/CVPR.2018.00745","article-title":"\u201cSqueeze-and-excitation networks,\u201d","volume-title":"2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Hu","year":"2018"},{"key":"B11","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1093\/acprof:oso\/9780195387247.003.0001","article-title":"\u201cIdeal-observer models of cue integration,\u201d","author":"Landy","year":"2011","journal-title":"Sensory Cue Integration"},{"key":"B12","doi-asserted-by":"crossref","first-page":"7772","DOI":"10.1109\/ICRA.2018.8460495","article-title":"\u201cSlip detection with combined tactile and visual information,\u201d","volume-title":"2018 IEEE International Conference on Robotics and Automation (ICRA)","author":"Li","year":"2018"},{"key":"B13","doi-asserted-by":"publisher","first-page":"9297","DOI":"10.1109\/TIE.2023.3323695","article-title":"Enhancing robotic tactile exploration with multireceptive graph convolutional networks","volume":"71","author":"Liao","year":"2024","journal-title":"IEEE Trans. Ind. Electron"},{"key":"B14","first-page":"737","article-title":"\u201cA self-supervised contrastive learning method for grasp outcomes prediction,\u201d","volume-title":"2023 IEEE International Conference on Real-time Computing and Robotics (RCAR)","author":"Liu","year":""},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TIM.2023.3285986","article-title":"A deep learning method based on triplet network using self-attention for tactile grasp outcomes prediction","volume":"72","author":"Liu","year":"","journal-title":"IEEE Trans. Instrum. Meas"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1708.04436","article-title":"Iterative closest labeled point for tactile object shape recognition","author":"Luo","year":"2017","journal-title":"arXiv"},{"key":"B17","doi-asserted-by":"publisher","first-page":"eaau4984","DOI":"10.1126\/scirobotics.aau4984","article-title":"Learning ambidextrous robot grasping policies","volume":"4","author":"Mahler","year":"2019","journal-title":"Sci. Robot"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.09611","article-title":"Mm1: Methods, analysis &insights from multimodal llm pre-training","author":"McKinzie","year":"2024","journal-title":"arXiv"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.07143","article-title":"Leave no context behind: efficient infinite context transformers with infini-attention","author":"Munkhdalai","year":"2024","journal-title":"arXiv"},{"key":"B20","doi-asserted-by":"publisher","first-page":"4435","DOI":"10.1109\/JSEN.2017.2707340","article-title":"Design of ultrawideband stepped-frequency radar for imaging of obscured targets","volume":"17","author":"Phelan","year":"2017","journal-title":"IEEE Sens. J"},{"key":"B21","doi-asserted-by":"crossref","first-page":"10248","DOI":"10.1109\/ICRA46639.2022.9811898","article-title":"\u201cPouring by feel: an analysis of tactile and proprioceptive sensing for accurate pouring,\u201d","volume-title":"2022 International Conference on Robotics and Automation (ICRA)","author":"Piacenza","year":"2022"},{"key":"B22","doi-asserted-by":"publisher","first-page":"e2106235118","DOI":"10.1073\/pnas.2106235118","article-title":"How multisensory neurons solve causal inference","volume":"118","author":"Rideaux","year":"2021","journal-title":"Proc. Natl. Acad. Sci"},{"key":"B23","first-page":"14663","article-title":"\u201cEgocentric auditory attention localization in conversations,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Ryan","year":"2023"},{"key":"B24","doi-asserted-by":"crossref","first-page":"13438","DOI":"10.1109\/ICRA48506.2021.9561877","article-title":"\u201cContact-graspnet: efficient 6-dof grasp generation in cluttered scenes,\u201d","volume-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)","author":"Sundermeyer","year":"2021"},{"key":"B25","first-page":"7872","article-title":"\u201cDeep multi-sensory object category recognition using interactive behavioral exploration,\u201d","volume-title":"International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24","author":"Tatiya","year":"2019"},{"key":"B26","article-title":"\u201cAttention is all you need,\u201d","volume-title":"Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"B27","doi-asserted-by":"publisher","first-page":"20694","DOI":"10.1109\/JSEN.2021.3100149","article-title":"Status identification and object in-hand reorientation using force\/torque sensors","volume":"21","author":"Wang","year":"2021","journal-title":"IEEE Sens. J"},{"key":"B28","doi-asserted-by":"publisher","first-page":"11426","DOI":"10.1109\/LRA.2022.3199684","article-title":"Visual haptic reasoning: estimating contact forces by observing deformable object interactions","volume":"7","author":"Wang","year":"2022","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B29","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1007\/978-3-319-03017-3_19","author":"Wettels","year":"2014","journal-title":"Multimodal Tactile Sensor"},{"key":"B30","doi-asserted-by":"publisher","first-page":"129201","DOI":"10.1007\/s11432-021-3512-6","article-title":"Robotic haptic adjective perception based on coupled sparse coding","volume":"66","author":"Xiong","year":"","journal-title":"Sci. China Inf. Sci"},{"key":"B31","doi-asserted-by":"publisher","first-page":"2259","DOI":"10.1109\/TII.2022.3195171","article-title":"Deeply supervised subspace learning for cross-modal material perception of known and unknown objects","volume":"19","author":"Xiong","year":"","journal-title":"IEEE Trans. Ind. Inf"},{"key":"B32","doi-asserted-by":"publisher","first-page":"6119","DOI":"10.1109\/TSMC.2023.3279023","article-title":"Robotic object perception based on multispectral few-shot coupled learning","volume":"53","author":"Xiong","year":"","journal-title":"IEEE Trans. Syst. Man Cybern. Syst"},{"key":"B33","article-title":"\u201cIngesTables: scalable and efficient training of LLM-enabled tabular foundation models,\u201d","volume-title":"NeurIPS 2023 Second Table Representation Learning Workshop","author":"Yak","year":"2023"},{"key":"B34","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1016\/j.neucom.2021.06.009","article-title":"Context-aware self-attention networks for natural language processing","volume":"458","author":"Yang","year":"2021","journal-title":"Neurocomputing"},{"key":"B35","article-title":"\u201cTouch and go: learning from human-collected vision and touch,\u201d","author":"Yang","year":"2022","journal-title":"Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022"},{"key":"B36","doi-asserted-by":"publisher","first-page":"959","DOI":"10.1109\/TASE.2021.3055251","article-title":"Touch modality identification with tensorial tactile signals: a kernel-based approach","volume":"19","author":"Yi","year":"2022","journal-title":"IEEE Trans. Autom. Sci. Eng"},{"key":"B37","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.01009","article-title":"\u201cExploring self-attention for image recognition,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Zhao","year":"2020"},{"key":"B38","first-page":"233","article-title":"\u201cVisual-haptic-kinesthetic object recognition with multimodal transformer,\u201d","volume-title":"International Conference on Artificial Neural Networks","author":"Zhou","year":"2023"},{"key":"B39","first-page":"10323","article-title":"\u201cBiformer: vision transformer with bi-level routing attention,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Zhu","year":"2023"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1484751\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,6]],"date-time":"2024-11-06T06:12:36Z","timestamp":1730873556000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2024.1484751\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,6]]},"references-count":39,"alternative-id":["10.3389\/fnbot.2024.1484751"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2024.1484751","relation":{},"ISSN":["1662-5218"],"issn-type":[{"type":"electronic","value":"1662-5218"}],"subject":[],"published":{"date-parts":[[2024,11,6]]},"article-number":"1484751"}}