{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T15:10:22Z","timestamp":1784387422988,"version":"3.55.0"},"reference-count":35,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T00:00:00Z","timestamp":1768953600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Joint Fund of the Zhejiang Provincial National Science Foundation of China","award":["LLSSZ25F030002"],"award-info":[{"award-number":["LLSSZ25F030002"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the advancement of tactile sensors, researchers increasingly integrate tactile perception into robotics, but only for tasks such as object reconstruction, classification, recognition, and grasp state assessment. In this paper, we rethink the relationship between visual and tactile perception and propose a novel robotic grasping detection method based on visual\u2013tactile perception. Initially, we construct a visual\u2013tactile dataset containing the grasp stability for each potential grasping position. Next, we introduce a novel Grasp Stability Prediction Module (GSPM) to generate a grasp stability probability map, providing prior knowledge regarding grasp stability to the grasp detection network for each possible grasp position. Finally, the map is multiplied element-wise with the corresponding colored image and inputted into the grasp detection network. Experimental results demonstrate that our novel visual\u2013tactile fusion method significantly enhances robotic grasping detection accuracy.<\/jats:p>","DOI":"10.3390\/s26020724","type":"journal-article","created":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T15:31:46Z","timestamp":1769009506000},"page":"724","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Enhancing Robotic Grasping Detection Using Visual\u2013Tactile Fusion Perception"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9258-8038","authenticated-orcid":false,"given":"Dongyuan","family":"Zheng","sequence":"first","affiliation":[{"name":"School of Artificial Intelligence, Lishui University, Lishui 323000, China"},{"name":"School of Computer Science, Central China Normal University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yahong","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Lishui University, Lishui 323000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"3300","DOI":"10.1109\/LRA.2018.2852779","article-title":"More than a feeling: Learning to grasp and regrasp using vision and touch","volume":"3","author":"Calandra","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"5827","DOI":"10.1109\/LRA.2020.3010720","article-title":"Self-attention based visual-tactile fusion learning for predicting grasp outcomes","volume":"5","author":"Cui","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"12235","DOI":"10.1109\/LRA.2022.3214785","article-title":"Symmetrygrasp: Symmetry-aware antipodal grasp detection from single-view RGB-D images","volume":"7","author":"Shi","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Patten, T., Park, K., Leitner, M., Wolfram, K., and Vincze, M. (2021). Object learning for 6D pose estimation and grasping from RGB-D videos of in-hand manipulation. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS51168.2021.9635884"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3838","DOI":"10.1109\/TRO.2023.3286071","article-title":"Visual\u2013Tactile Fusion for Transparent Object Grasping in Complex Backgrounds","volume":"39","author":"Li","year":"2023","journal-title":"IEEE Trans. Robot."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"5384","DOI":"10.1109\/LRA.2023.3295296","article-title":"Visual-tactile robot grasping based on human skill learning from demonstrations using a wearable parallel hand exoskeleton","volume":"8","author":"Lu","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Cui, S., Wang, R., Wei, J., Li, F., and Wang, S. (2020). Grasp state assessment of deformable objects using visual-tactile fusion perception. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA40945.2020.9196787"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"102601","DOI":"10.1016\/j.rcim.2023.102601","article-title":"Digital twin-enabled grasp outcomes assessment for unknown objects using visual-tactile fusion perception","volume":"84","author":"Zhang","year":"2023","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tahoun, M., Tahri, O., Ram\u00f3n, J.A.C., and Mezouar, Y. (2021). Visual-Tactile Fusion for 3D Objects Reconstruction from a Single Depth View and a Single Gripper Touch for Robotics Tasks. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS51168.2021.9636150"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Xu, W., Yu, Z., Xue, H., Ye, R., Yao, S., and Lu, C. (2023, January 18\u201322). Visual-Tactile Sensing for In-Hand Object Reconstruction. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00850"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"850","DOI":"10.1109\/TCDS.2023.3311871","article-title":"A Human-Like Siamese-Based Visual-Tactile Fusion Model for Object Recognition","volume":"16","author":"Wang","year":"2023","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"4349","DOI":"10.1109\/TSMC.2021.3096235","article-title":"Partial visual-tactile fused learning for robotic object recognition","volume":"52","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Syst. Man, Cybern. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"5238","DOI":"10.1109\/LRA.2022.3145064","article-title":"SE-ResUNet: A novel robotic grasp detection method","volume":"7","author":"Yu","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Morrison, D., Corke, P., and Leitner, J. (2018). Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach. arXiv.","DOI":"10.15607\/RSS.2018.XIV.021"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"8541","DOI":"10.1109\/LRA.2023.3330664","article-title":"VERGNet: Visual Enhancement Guided Robotic Grasp Detection under Low-light Condition","volume":"8","author":"Niu","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Depierre, A., Dellandr\u00e9a, E., and Chen, L. (2018). Jacquard: A large scale dataset for robotic grasp detection. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS.2018.8593950"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1177\/0278364914549607","article-title":"Deep learning for detecting robotic grasps","volume":"34","author":"Lenz","year":"2015","journal-title":"Int. J. Robot. Res."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fang, H.S., Wang, C., Gou, M., and Lu, C. (2020, January 14\u201319). Graspnet-1billion: A large-scale benchmark for general object grasping. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01146"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Brahmbhatt, S., Ham, C., Kemp, C.C., and Hays, J. (2019, January 15\u201320). Contactdb: Analyzing and predicting grasp contact via thermal imaging. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00891"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Vuong, A.D., Vu, M.N., Le, H., Huang, B., Huynh, B., Vo, T., Kugi, A., and Nguyen, A. (2023). Grasp-anything: Large-scale grasp dataset from foundation models. arXiv.","DOI":"10.1109\/ICRA57147.2024.10611277"},{"key":"ref_21","unstructured":"Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., and Brohan, A. (2023). Open x-embodiment: Robotic learning datasets and rt-x models. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2974","DOI":"10.1109\/TASE.2023.3272664","article-title":"FANet: Fast and Accurate Robotic Grasp Detection Based on Keypoints","volume":"21","author":"Zhai","year":"2023","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2683","DOI":"10.1109\/TIE.2023.3270537","article-title":"Information-Theoretic Exploration for Adaptive Robotic Grasping in Clutter Based on Real-Time Pixel-Level Grasp Detection","volume":"71","author":"Wu","year":"2023","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Huang, H., Wang, D., Zhu, X., Walters, R., and Platt, R. (2023). Edge grasp network: A graph-based se (3)-invariant approach to grasp detection. Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE.","DOI":"10.1109\/ICRA48891.2023.10160728"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3130","DOI":"10.1109\/TASE.2023.3275771","article-title":"Anchor-Based Multi-Scale Deep Grasp Pose Detector With Encoded Angle Regression","volume":"21","author":"Cheng","year":"2023","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Yuan, W., Dong, S., and Adelson, E.H. (2017). Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17.","DOI":"10.3390\/s17122762"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3838","DOI":"10.1109\/LRA.2020.2977257","article-title":"Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation","volume":"5","author":"Lambeta","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1089\/soro.2017.0052","article-title":"The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies","volume":"5","author":"Pestell","year":"2018","journal-title":"Soft Robot."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"6307","DOI":"10.1109\/LRA.2023.3306670","article-title":"Tirgel: A visuo-tactile sensor with total internal reflection mechanism for external observation and contact detection","volume":"8","author":"Zhang","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"5512","DOI":"10.1109\/LRA.2023.3296371","article-title":"Visual-tactile learning of garment unfolding for robot-assisted dressing","volume":"8","author":"Zhang","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Zhou, Z., Wang, H., Zhang, Z., Huang, H., and Cao, Q. (2023). Grasp Stability Assessment Through Attention-Guided Cross-Modality Fusion and Transfer Learning. Proceedings of the 2023 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS55552.2023.10342411"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1106","DOI":"10.1109\/LRA.2023.3337690","article-title":"Enhancing generalizable 6d pose tracking of an in-hand object with tactile sensing","volume":"9","author":"Liu","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"108176","DOI":"10.1016\/j.patcog.2021.108176","article-title":"Lifelong robotic visual-tactile perception learning","volume":"121","author":"Dong","year":"2022","journal-title":"Pattern Recognit."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"11611","DOI":"10.1109\/TIE.2021.3120474","article-title":"High-performance pixel-level grasp detection based on adaptive grasping and grasp-aware network","volume":"69","author":"Wang","year":"2021","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kumra, S., Joshi, S., and Sahin, F. (2020). Antipodal robotic grasping using generative residual convolutional neural network. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE.","DOI":"10.1109\/IROS45743.2020.9340777"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/2\/724\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T15:34:55Z","timestamp":1769009695000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/2\/724"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,21]]},"references-count":35,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["s26020724"],"URL":"https:\/\/doi.org\/10.3390\/s26020724","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,21]]}}}