{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,6]],"date-time":"2026-11-06T01:36:57Z","timestamp":1793929017016,"version":"4.8.0"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T00:00:00Z","timestamp":1687219200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>The use of robots in various industries is evolving from mechanization to intelligence and precision. These systems often comprise parts made of different materials and thus require accurate and comprehensive target identification. While humans perceive the world through a highly diverse perceptual system and can rapidly identify deformable objects through vision and touch to prevent slipping or excessive deformation during grasping, robot recognition technology mainly relies on visual sensors, which lack critical information such as object material, leading to incomplete cognition. Therefore, multimodal information fusion is believed to be key to the development of robot recognition. Firstly, a method of converting tactile sequences to images is proposed to deal with the obstacles of information exchange between different modalities for vision and touch, which overcomes the problems of the noise and instability of tactile data. Subsequently, a visual-tactile fusion network framework based on an adaptive dropout algorithm is constructed, together with an optimal joint mechanism between visual information and tactile information established, to solve the problem of mutual exclusion or unbalanced fusion in traditional fusion methods. Finally, experiments show that the proposed method effectively improves robot recognition ability, and the classification accuracy is as high as 99.3%.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1181383","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T02:27:11Z","timestamp":1687228031000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Adaptive visual\u2013tactile fusion recognition for robotic operation of multi-material system"],"prefix":"10.3389","volume":"17","author":[{"given":"Zihao","family":"Ding","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guodong","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhenhua","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lining","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/j.inffus.2022.11.032","article-title":"Fusion of tactile and visual information in deep learning models for object recognition","volume":"92","author":"Babadian","year":"2023","journal-title":"Informat. Fusion"},{"key":"B2","doi-asserted-by":"publisher","first-page":"3300","DOI":"10.1109\/LRA.2018.2852779","article-title":"More than a feeling: learning to grasp and regrasp using vision and touch","volume":"3","author":"Calandra","year":"2018","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B3","doi-asserted-by":"publisher","first-page":"3427","DOI":"10.1109\/LRA.2022.3146565","article-title":"Using collocated vision and tactile sensors for visual servoing and localization","volume":"7","author":"Chaudhury","year":"2022","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B4","doi-asserted-by":"publisher","first-page":"3355","DOI":"10.1109\/LRA.2018.2852777","article-title":"Real-world multiobject, multigrasp detection","volume":"3","author":"Chu","year":"2018","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B5","doi-asserted-by":"publisher","first-page":"538","DOI":"10.1109\/ICRA40945.2020.9196787","article-title":"\u201cGrasp state assessment of deformable objects using visual-tactile fusion perception,\u201d","author":"Cui","year":"2020","journal-title":"2020 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B6","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1007\/s10479-005-5724-z","article-title":"A tutorial on the cross-entropy method","volume":"134","author":"De Boer","year":"2005","journal-title":"Ann. Operat. Res."},{"key":"B7","first-page":"52","article-title":"\u201cDoes object recognition work for everyone?,\u201d","author":"De Vries","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, arXiv"},{"key":"B8","doi-asserted-by":"publisher","first-page":"541","DOI":"10.3390\/e21060541","article-title":"Approximate entropy and sample entropy: a comprehensive tutorial","volume":"21","author":"Delgado-Bonal","year":"2019","journal-title":"Entropy"},{"key":"B9","doi-asserted-by":"publisher","first-page":"303","DOI":"10.3389\/fnbot.2022.1120167","article-title":"Toward and beyond human-level AI, volume II","volume":"16","author":"Dunin-Barkowski","year":"2023","journal-title":"Front. Neurorobot."},{"key":"B10","first-page":"1050","article-title":"\u201cDropout as a bayesian approximation: representing model uncertainty in deep learning,\u201d","author":"Gal","year":"2016","journal-title":"International Conference on Machine Learning: PMLR"},{"key":"B11","doi-asserted-by":"publisher","first-page":"2100074","DOI":"10.1002\/aisy.202100074","article-title":"Tactile and vision perception for intelligent humanoids","volume":"4","author":"Gao","year":"2022","journal-title":"Adv. Intell. Syst."},{"key":"B12","doi-asserted-by":"publisher","first-page":"536","DOI":"10.1109\/ICRA.2016.7487176","article-title":"\u201cDeep learning for tactile understanding from visual and haptic data,\u201d","author":"Gao","year":"2016","journal-title":"2016 IEEE International Conference on Robotics and Automation (ICRA)"},{"key":"B13","author":"Grimaldi","year":"2022","journal-title":"A Robust Event-Driven Approach to Always-on Object Recognition, IEEE"},{"key":"B14","doi-asserted-by":"publisher","first-page":"24","DOI":"10.3389\/fnbot.2023.1093132","article-title":"Viewpoint planning with transition management for active object recognition","volume":"17","author":"Haibo","year":"2023","journal-title":"Front. Neurorobot."},{"key":"B15","doi-asserted-by":"publisher","first-page":"3215749","DOI":"10.1109\/TMRB.2022.3215749","article-title":"Review of bioinspired vision-tactile fusion perception (VTFP): From humans to humanoids","volume":"2022","author":"He","year":"2022","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"B16","doi-asserted-by":"publisher","first-page":"562","DOI":"10.1109\/TOH.2019.2959993","article-title":"Tactile roughness perception of virtual gratings by electrovibration","volume":"13","author":"I\u015fleyen","year":"2019","journal-title":"IEEE Trans. Hapt."},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1704.08863","article-title":"On weight initialization in deep neural networks","author":"Kumar","year":"2017","journal-title":"arXiv preprint arXiv:1704.08863"},{"key":"B18","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B19","doi-asserted-by":"publisher","first-page":"8943","DOI":"10.1109\/ICRA.2019.8793485","article-title":"\u201cMaking sense of vision and touch: self-supervised learning of multimodal representations for contact-rich tasks,\u201d","author":"Lee","year":"2019","journal-title":"2019 International Conference on Robotics and Automation (ICRA)"},{"key":"B20","first-page":"10609","article-title":"\u201cConnecting touch and vision via cross-modal prediction,\u201d","author":"Li","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B21","doi-asserted-by":"publisher","first-page":"2466","DOI":"10.1109\/TSMC.2018.2818184","article-title":"Cross-modal zero-shot-learning for tactile object recognition","volume":"50","author":"Liu","year":"","journal-title":"IEEE Trans. Syst. Man Cybernet."},{"key":"B22","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1109\/TCDS.2018.2819826","article-title":"Active visual-tactile cross-modal matching","volume":"11","author":"Liu","year":"","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"B23","doi-asserted-by":"publisher","first-page":"996","DOI":"10.1109\/TASE.2016.2549552","article-title":"Visual\u2013tactile fusion for object recognition","volume":"14","author":"Liu","year":"2016","journal-title":"IEEE Trans. Automat. Sci. Eng."},{"key":"B24","author":"Nixon","year":"2019","journal-title":"Feature Extraction and Image Processing for Computer Vision."},{"key":"B25","doi-asserted-by":"publisher","first-page":"1303","DOI":"10.1007\/s10846-020-01303-z","article-title":"Towards autonomous robotic assembly: using combined visual and tactile sensing for adaptive task execution","volume":"101","author":"Nottensteiner","year":"2021","journal-title":"J. Intell. Robot. Syst."},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3449065","article-title":"Effects of tactile textures on preference in visuo-tactile exploration","volume":"18","author":"Park","year":"2021","journal-title":"ACM Trans. Appl. Percept."},{"key":"B27","doi-asserted-by":"publisher","first-page":"102053","DOI":"10.1016\/j.displa.2021.102053","article-title":"Review of multi-view 3D object recognition methods based on deep learning","volume":"69","author":"Qi","year":"2021","journal-title":"Displays"},{"key":"B28","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1016\/S0076-6879(04)84011-4","article-title":"Sample entropy","volume":"384","author":"Richman","year":"2004","journal-title":"Methods Enzymol."},{"key":"B29","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1007\/978-981-99-0617-8_43","article-title":"\u201cVision-tactile fusion based detection of deformation and slippage of deformable objects during grasping,\u201d","author":"Ruan","year":"2023","journal-title":"Cognitive Systems and Information Processing: 7th International Conference, ICCSIP 2022, Fuzhou, China, December 17-18, 2022, Revised Selected Papers"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10846-021-01431-0","article-title":"On the design and development of vision-based tactile sensors","volume":"102","author":"Shah","year":"2021","journal-title":"J. Intell. Robot. Syst."},{"key":"B31","doi-asserted-by":"publisher","first-page":"200","DOI":"10.1016\/j.patrec.2022.11.023","article-title":"CNN-transformer for visual-tactile fusion applied in road recognition of autonomous vehicles","volume":"166","author":"Shi","year":"2023","journal-title":"Patt. Recogn. Lett."},{"key":"B32","doi-asserted-by":"publisher","first-page":"969","DOI":"10.1109\/TSMC.2016.2524059","article-title":"Object classification and grasp planning using visual and tactile sensing","volume":"46","author":"Sun","year":"2016","journal-title":"IEEE Trans. Syst. Man Cybernet."},{"key":"B33","doi-asserted-by":"publisher","first-page":"9035","DOI":"10.1109\/ICRA.2019.8793502","article-title":"\u201cRobust learning of tactile force estimation through robot interaction,\u201d","author":"Sundaralingam","year":"2019","journal-title":"2019 International Conference on Robotics and Automation (ICRA)"},{"key":"B34","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1016\/j.sna.2019.02.032","article-title":"Using a convolutional neural network to construct a pen-type tactile sensor system for roughness recognition","volume":"291","author":"Tsuji","year":"2019","journal-title":"Sensor. Actuat. A"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1606","DOI":"10.1109\/IROS.2018.8593430","article-title":"\u201c3d shape perception from monocular vision, touch, and shape priors,\u201d","author":"Wang","year":"2018","journal-title":"2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)"},{"key":"B36","doi-asserted-by":"publisher","first-page":"9087","DOI":"10.1109\/ICRA.2019.8793842","article-title":"\u201cEagle Shoal: a new designed modular tactile sensing dexterous hand for domestic service robots,\u201d","author":"Wang","year":"2019","journal-title":"2019 International Conference on Robotics and Automation (ICRA)"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1108\/IR-12-2021-0303","article-title":"DoraHand: a novel dexterous hand with tactile sensing finger module","author":"Wang","year":"2022","journal-title":"Industr. Robot"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1109\/TMECH.2021.3080378","article-title":"Human exploratory procedures based hybrid measurement fusion for material recognition","author":"Xiong","year":"2021","journal-title":"IEEE\/ASME Trans. Mechatr"},{"key":"B39","first-page":"169","article-title":"\u201cProcess reliability improvement technology of micro high precision fiber coil for integrated fiber optic gyroscope,\u201d","author":"Yan","year":"2023","journal-title":"AOPC 2022: Optoelectronics and Nanophotonics: SPIE"},{"key":"B40","doi-asserted-by":"publisher","first-page":"2398","DOI":"10.1109\/TSMC.2017.2676022","article-title":"Haptic identification by ELM-controlled uncertain manipulator","volume":"47","author":"Yang","year":"2017","journal-title":"IEEE Trans. Syst. Man Cybernet."},{"key":"B41","first-page":"5580","article-title":"\u201cConnecting look and feel: associating the visual and tactile properties of physical materials,\u201d","author":"Yuan","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, arXiv"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.2021.3096235","article-title":"Partial visual-tactile fused learning for robotic object recognition","author":"Zhang","year":"","journal-title":"IEEE Trans. Syst. Man Cybernet"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2021.3080321","article-title":"Visual-tactile fused graph learning for object clustering","author":"Zhang","year":"","journal-title":"IEEE Trans. Cybernet"},{"key":"B44","doi-asserted-by":"publisher","first-page":"10426","DOI":"10.1609\/aaai.v34i06.6612","article-title":"\u201cVisual tactile fusion object clustering,\u201d","author":"Zhang","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B45","doi-asserted-by":"publisher","first-page":"2407","DOI":"10.1109\/TMM.2016.2598140","article-title":"Deep learning for surface material classification using haptic and visual information","volume":"18","author":"Zheng","year":"2016","journal-title":"IEEE Trans. Multimedia"},{"key":"B46","doi-asserted-by":"publisher","first-page":"1192","DOI":"10.1109\/TNNLS.2020.2980892","article-title":"Lifelong visual-tactile cross-modal learning for robotic material perception","volume":"32","author":"Zheng","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1181383\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T02:27:21Z","timestamp":1687228041000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1181383\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,20]]},"references-count":46,"alternative-id":["10.3389\/fnbot.2023.1181383"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1181383","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,20]]},"article-number":"1181383"}}