{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T16:29:53Z","timestamp":1778171393450,"version":"3.51.4"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T00:00:00Z","timestamp":1754006400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T00:00:00Z","timestamp":1754006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"HORIZON EUROPE Research and Innovation Actions","award":["101070310"],"award-info":[{"award-number":["101070310"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Intel Serv Robotics"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Combining visual and tactile cues has proven effective for object recognition, grasping, and manipulation tasks. However, integrating these modalities is challenging as tactile and visual data convey distinct information and differ structurally. Researchers have addressed this problem by proposing approaches that either do not consider mechanical properties conveyed by tactile sensors or cannot be deployed directly for diverse tasks. In this paper, we propose a cross-task visuo-tactile representation that encodes both the geometrical and mechanical properties of objects in a point cloud (PC) data structure. By physically exploring different areas of a given item, we collect tactile information to estimate the local compliance of the surface, encoding it as the color information of the PC in the probed areas. This color information is extended to the entire object assuming that neighboring points share the same mechanical properties. We apply the proposed PC to six real-world objects showing that it can be effectively used to encode their shape along with their information on the local compliance. Further, we show that the augmented PC can be used for different tasks by exploiting this in three robotic tasks\u2014a visuo-tactile object classification, a path following and a reaching in clutter.<\/jats:p>","DOI":"10.1007\/s11370-025-00628-8","type":"journal-article","created":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T09:31:08Z","timestamp":1754040668000},"page":"1119-1131","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A cross-task visuo-tactile representation using point clouds"],"prefix":"10.1007","volume":"18","author":[{"given":"Giammarco","family":"Caroleo","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alessandro","family":"Albini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Perla","family":"Maiolino","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,8,1]]},"reference":[{"key":"628_CR1","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1016\/j.neuroimage.2014.04.024","volume":"97","author":"E Porcu","year":"2014","unstructured":"Porcu E, Keitel C, M\u00fcller MM (2014) Visual, auditory and tactile stimuli compete for early sensory processing capacities within but not between senses. Neuroimage 97:224\u2013235","journal-title":"Neuroimage"},{"issue":"3","key":"628_CR2","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1109\/MRA.2022.3212316","volume":"30","author":"C Parsons","year":"2022","unstructured":"Parsons C, Albini A, De Martini D, Maiolino P (2022) Visuo-tactile recognition of partial point clouds using pointnet and curriculum learning: enabling tactile perception from visual data. IEEE Robot Autom Magaz 30(3):69\u201378","journal-title":"IEEE Robot Autom Magaz"},{"key":"628_CR3","doi-asserted-by":"crossref","unstructured":"Falco P,\u00a0Lu S,\u00a0Cirillo A,\u00a0Natale C,\u00a0Pirozzi S,\u00a0Lee D (2017) \u201cCross-modal visuo-tactile object recognition using robotic active exploration,\u201d in 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 5273\u20135280","DOI":"10.1109\/ICRA.2017.7989619"},{"issue":"4","key":"628_CR4","doi-asserted-by":"publisher","first-page":"9557","DOI":"10.1109\/LRA.2022.3191408","volume":"7","author":"PK Murali","year":"2022","unstructured":"Murali PK, Wang C, Lee D, Dahiya R, Kaboli M (2022) Deep active cross-modal visuo-tactile transfer learning for robotic object recognition. IEEE Robot Autom Lett 7(4):9557\u20139564","journal-title":"IEEE Robot Autom Lett"},{"key":"628_CR5","doi-asserted-by":"crossref","unstructured":"Wei F,\u00a0Zhao J,\u00a0Shan C,\u00a0Yuan Z (2022) \u201cAlignment and multi-scale fusion for visual-tactile object recognition,\u201d in 2022 International joint conference on neural networks (IJCNN). IEEE, 1\u20138","DOI":"10.1109\/IJCNN55064.2022.9892134"},{"key":"628_CR6","doi-asserted-by":"crossref","unstructured":"Bj\u00f6rkman M,\u00a0Bekiroglu Y,\u00a0H\u00f6gman V,\u00a0Kragic D (2013) \u201cEnhancing visual perception of shape through tactile glances,\u201d in 2013 IEEE\/RSJ International conference on intelligent robots and systems. IEEE, 3180\u20133186","DOI":"10.1109\/IROS.2013.6696808"},{"key":"628_CR7","first-page":"14193","volume":"33","author":"E Smith","year":"2020","unstructured":"Smith E, Calandra R, Romero A, Gkioxari G, Meger D, Malik J, Drozdzal M (2020) 3d shape reconstruction from vision and touch. Adv Neural Inf Process Syst 33:14193\u201314206","journal-title":"Adv Neural Inf Process Syst"},{"key":"628_CR8","first-page":"16064","volume":"34","author":"E Smith","year":"2021","unstructured":"Smith E, Meger D, Pineda L, Calandra R, Malik J, Romero\u00a0Soriano A, Drozdzal M (2021) Active 3d shape reconstruction from vision and touch. Adv Neural Inf Process Syst 34:16064\u201316078","journal-title":"Adv Neural Inf Process Syst"},{"key":"628_CR9","doi-asserted-by":"crossref","unstructured":"Suresh S,\u00a0Si Z, Mangelson JG,\u00a0Yuan W,\u00a0Kaess M (2022) \u201cShapemap 3-d: Efficient shape mapping through dense touch and vision,\u201d in 2022 International conference on robotics and automation (ICRA). IEEE, 7073\u20137080","DOI":"10.1109\/ICRA46639.2022.9812040"},{"key":"628_CR10","doi-asserted-by":"crossref","unstructured":"Bimbo J, Seneviratne LD,\u00a0Althoefer K,\u00a0Liu H (2013) \u201cCombining touch and vision for the estimation of an object\u2019s pose during manipulation,\u201d in 2013 IEEE\/RSJ international conference on intelligent robots and systems. IEEE, 4021\u20134026","DOI":"10.1109\/IROS.2013.6696931"},{"key":"628_CR11","doi-asserted-by":"crossref","unstructured":"Pecyna L,\u00a0Dong S,\u00a0Luo S (2022) \u201cVisual-tactile multimodality for following deformable linear objects using reinforcement learning,\u201d in 2022 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, 3987\u20133994","DOI":"10.1109\/IROS47612.2022.9982218"},{"key":"628_CR12","doi-asserted-by":"crossref","unstructured":"Ottenhaus S,\u00a0Renninghoff D,\u00a0Grimm R,\u00a0Ferreira F,\u00a0Asfour T (2019) \u201cVisuo-haptic grasping of unknown objects based on gaussian process implicit surfaces and deep learning,\u201d in 2019 IEEE-RAS 19th international conference on humanoid robots (Humanoids). IEEE, pp. 402\u2013409","DOI":"10.1109\/Humanoids43949.2019.9035002"},{"issue":"4","key":"628_CR13","doi-asserted-by":"publisher","first-page":"987","DOI":"10.1109\/TRO.2019.2914772","volume":"35","author":"P Falco","year":"2019","unstructured":"Falco P, Lu S, Natale C, Pirozzi S, Lee D (2019) A transfer learning approach to cross-modal object recognition: from visual observation to robotic haptic exploration. IEEE Trans Rob 35(4):987\u2013998","journal-title":"IEEE Trans Rob"},{"issue":"2","key":"628_CR14","doi-asserted-by":"publisher","first-page":"4686","DOI":"10.1109\/LRA.2022.3150045","volume":"7","author":"PK Murali","year":"2022","unstructured":"Murali PK, Dutta A, Gentner M, Burdet E, Dahiya R, Kaboli M (2022) Active visuo-tactile interactive robotic perception for accurate object pose estimation in dense clutter. IEEE Robot Autom Lett 7(2):4686\u20134693","journal-title":"IEEE Robot Autom Lett"},{"key":"628_CR15","unstructured":"Villalonga MB,\u00a0Rodriguez A,\u00a0Lim B,\u00a0Valls E,\u00a0Sechopoulos T (2021) \u201cTactile object pose estimation from the first touch with geometric contact rendering,\u201d in Conference on robot learning. PMLR, 1015\u20131029"},{"key":"628_CR16","doi-asserted-by":"crossref","unstructured":"Dragiev S,\u00a0Toussaint M,\u00a0Gienger M (2011) \u201cGaussian process implicit surfaces for shape estimation and grasping,\u201d in 2011 IEEE international conference on robotics and automation. IEEE, pp. 2845\u20132850","DOI":"10.1109\/ICRA.2011.5980395"},{"issue":"2","key":"628_CR17","doi-asserted-by":"publisher","first-page":"5254","DOI":"10.1109\/LRA.2022.3152975","volume":"7","author":"L Rustler","year":"2022","unstructured":"Rustler L, Lundell J, Behrens JK, Kyrki V, Hoffmann M (2022) Active visuo-haptic object shape completion. IEEE Robot Autom Lett 7(2):5254\u20135261","journal-title":"IEEE Robot Autom Lett"},{"issue":"2","key":"628_CR18","doi-asserted-by":"publisher","first-page":"996","DOI":"10.1109\/TASE.2016.2549552","volume":"14","author":"H Liu","year":"2016","unstructured":"Liu H, Yu Y, Sun F, Gu J (2016) Visual-tactile fusion for object recognition. IEEE Trans Autom Sci Eng 14(2):996\u20131008","journal-title":"IEEE Trans Autom Sci Eng"},{"key":"628_CR19","doi-asserted-by":"crossref","unstructured":"Bhattacharjee T, Shenoi AA,\u00a0Park D, Rehg JM, Kemp CC (2015) \u201cCombining tactile sensing and vision for rapid haptic mapping,\u201d in 2015 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp. 1200\u20131207","DOI":"10.1109\/IROS.2015.7353522"},{"key":"628_CR20","doi-asserted-by":"crossref","unstructured":"Shenoi AA,\u00a0Bhattacharjee T, Kemp CC (2016) \u201cA crf that combines touch and vision for haptic mapping,\u201d in 2016 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2255\u20132262","DOI":"10.1109\/IROS.2016.7759353"},{"issue":"2","key":"628_CR21","doi-asserted-by":"publisher","first-page":"818","DOI":"10.1109\/TCSVT.2022.3206865","volume":"33","author":"Y Liu","year":"2022","unstructured":"Liu Y, Cong Y, Sun G, Ding Z (2022) Lifelong visual-tactile spectral clustering for robotic object perception. IEEE Trans Circuit Syst Video Technol 33(2):818\u2013829","journal-title":"IEEE Trans Circuit Syst Video Technol"},{"key":"628_CR22","doi-asserted-by":"crossref","unstructured":"Yang F,\u00a0Feng C,\u00a0Chen Z,\u00a0Park H,\u00a0Wang D,\u00a0Dou Y,\u00a0Zeng Z,\u00a0Chen X,\u00a0Gangopadhyay R,\u00a0Owens A et\u00a0al (2024) \u201cBinding touch to everything: Learning unified multimodal tactile representations,\u201d In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, , pp. 26\u00a0340\u201326\u00a0353","DOI":"10.1109\/CVPR52733.2024.02488"},{"key":"628_CR23","doi-asserted-by":"crossref","unstructured":"Rosales C,\u00a0Ajoudani A,\u00a0Gabiccini M,\u00a0Bicchi A (2014) \u201cActive gathering of frictional properties from objects,\u201d in 2014 IEEE\/RSJ international conference on intelligent robots and systems. IEEE, pp. 3982\u20133987","DOI":"10.1109\/IROS.2014.6943122"},{"issue":"2","key":"628_CR24","doi-asserted-by":"publisher","first-page":"2838","DOI":"10.1109\/LRA.2021.3062585","volume":"6","author":"TN Le","year":"2021","unstructured":"Le TN, Verdoja F, Abu-Dakka FJ, Kyrki V (2021) Probabilistic surface friction estimation based on visual and haptic measurements. IEEE Robot Autom Lett 6(2):2838\u20132845","journal-title":"IEEE Robot Autom Lett"},{"key":"628_CR25","doi-asserted-by":"crossref","unstructured":"Yao S,\u00a0Hauser K (2023) \u201cEstimating tactile models of heterogeneous deformable objects in real time,\u201d in 2023 IEEE international conference on robotics and automation (ICRA), pp. 12\u00a0583\u201312\u00a0589","DOI":"10.1109\/ICRA48891.2023.10160731"},{"key":"628_CR26","unstructured":"Qi CR,\u00a0Su H,\u00a0Mo K, Guibas LJ (2017) \u201cPointnet: Deep learning on point sets for 3d classification and segmentation,\u201d in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652\u2013660"},{"issue":"1","key":"628_CR27","doi-asserted-by":"publisher","first-page":"113","DOI":"10.3390\/s21010113","volume":"21","author":"G Rouhafzay","year":"2020","unstructured":"Rouhafzay G, Cretu A-M, Payeur P (2020) Transfer of learning from vision to touch: a hybrid deep convolutional neural network for visuo-tactile 3d object recognition. Sensors 21(1):113","journal-title":"Sensors"},{"key":"628_CR28","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/j.inffus.2022.11.032","volume":"92","author":"RP Babadian","year":"2023","unstructured":"Babadian RP, Faez K, Amiri M, Falotico E (2023) Fusion of tactile and visual information in deep learning models for object recognition. Inf Fusion 92:313\u2013325","journal-title":"Inf Fusion"},{"key":"628_CR29","unstructured":"Williams C,\u00a0Rasmussen C (1995) \u201cGaussian processes for regression,\u201d Advances in neural information processing systems, 8"},{"key":"628_CR30","doi-asserted-by":"crossref","unstructured":"Caccamo S,\u00a0G\u00fcler P,\u00a0Kjellstr\u00f6m H,\u00a0Kragic D (2016) \u201cActive perception and modeling of deformable surfaces using gaussian processes and position-based dynamics,\u201d in 2016 IEEE-RAS 16th international conference on humanoid robots (Humanoids). IEEE, pp. 530\u2013537","DOI":"10.1109\/HUMANOIDS.2016.7803326"},{"key":"628_CR31","doi-asserted-by":"crossref","unstructured":"Salman H,\u00a0Ayvali E, Srivatsan RA,\u00a0Ma Y,\u00a0Zevallos N,\u00a0Yasin R,\u00a0Wang L,\u00a0Simaan N,\u00a0Choset H (2018) \u201cTrajectory-optimized sensing for active search of tissue abnormalities in robotic surgery,\u201d in 2018 IEEE international conference on robotics and automation (ICRA). IEEE, pp. 5356\u20135363","DOI":"10.1109\/ICRA.2018.8460936"},{"key":"628_CR32","doi-asserted-by":"crossref","unstructured":"Border R, Gammell JD,\u00a0Newman P (2018) \u201cSurface edge explorer (see): Planning next best views directly from 3d observations,\u201d in 2018 IEEE international conference on robotics and automation (ICRA). IEEE, pp. 6116\u20136123","DOI":"10.1109\/ICRA.2018.8461098"},{"key":"628_CR33","doi-asserted-by":"crossref","unstructured":"Kangro J,\u00a0Traversaro S,\u00a0Pucci D,\u00a0Nori F (2017) \u201cSkin normal force calibration using vacuum bags,\u201d in 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp. 401\u2013406","DOI":"10.1109\/ICRA.2017.7989052"},{"issue":"4","key":"628_CR34","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1007\/s00454-011-9360-x","volume":"46","author":"F Chazal","year":"2011","unstructured":"Chazal F, Guibas LJ, Oudot SY, Skraba P (2011) Scalar field analysis over point cloud data. Discret Comput Geomet 46(4):743\u2013775","journal-title":"Discret Comput Geomet"},{"key":"628_CR35","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/j.ins.2022.11.139","volume":"622","author":"AM Ikotun","year":"2023","unstructured":"Ikotun AM, Ezugwu AE, Abualigah L, Abuhaija B, Heming J (2023) K-means clustering algorithms: a comprehensive review, variants analysis, and advances in the era of big data. Inf Sci 622:178\u2013210","journal-title":"Inf Sci"},{"key":"628_CR36","unstructured":"\u201cCyskin,\u201d https:\/\/www.cyskin.com\/"},{"key":"628_CR37","doi-asserted-by":"crossref","unstructured":"Sadeghi Aval\u00a0Shahr M,\u00a0Abbasimoshaei A,\u00a0Schwartz C, Kern TA (2024) \u201cDesigning, fabricating, and analyzing the whisker sensor for autonomous surface defect detection,\u201d in EUROSENSORS XXXVI","DOI":"10.5162\/EUROSENSORSXXXVI\/PT1.225"},{"key":"628_CR38","doi-asserted-by":"crossref","unstructured":"Caroleo G, Giovinazzo F, Albini A, Grella F, Cannata G, Maiolino P (2024) \u201cA proxy-tactile reactive control for robots moving in clutter,\u201d in. IEEE\/RSJ international conference on intelligent robots and systems (IROS) 2024:733\u2013739","DOI":"10.1109\/IROS58592.2024.10802341"},{"issue":"3","key":"628_CR39","doi-asserted-by":"publisher","first-page":"4361","DOI":"10.1109\/LRA.2021.3068110","volume":"6","author":"A Albini","year":"2021","unstructured":"Albini A, Grella F, Maiolino P, Cannata G (2021) Exploiting distributed tactile sensors to drive a robot arm through obstacles. IEEE Robot Autom Lett 6(3):4361\u20134368","journal-title":"IEEE Robot Autom Lett"},{"issue":"7758","key":"628_CR40","doi-asserted-by":"publisher","first-page":"698","DOI":"10.1038\/s41586-019-1234-z","volume":"569","author":"S Sundaram","year":"2019","unstructured":"Sundaram S, Kellnhofer P, Li Y, Zhu J-Y, Torralba A, Matusik W (2019) Learning the signatures of the human grasp using a scalable tactile glove. Nature 569(7758):698\u2013702","journal-title":"Nature"},{"key":"628_CR41","doi-asserted-by":"crossref","unstructured":"Yuan W,\u00a0Wang S,\u00a0Dong S,\u00a0Adelson E (2017) \u201cConnecting look and feel: Associating the visual and tactile properties of physical materials,\u201d in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5580\u20135588","DOI":"10.1109\/CVPR.2017.478"},{"key":"628_CR42","doi-asserted-by":"crossref","unstructured":"Johnson MK, Adelson EH (2009) \u201cRetrographic sensing for the measurement of surface texture and shape,\u201d in 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp. 1070\u20131077","DOI":"10.1109\/CVPRW.2009.5206534"}],"container-title":["Intelligent Service Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11370-025-00628-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11370-025-00628-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11370-025-00628-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T03:48:42Z","timestamp":1763437722000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11370-025-00628-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,1]]},"references-count":42,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["628"],"URL":"https:\/\/doi.org\/10.1007\/s11370-025-00628-8","relation":{},"ISSN":["1861-2776","1861-2784"],"issn-type":[{"value":"1861-2776","type":"print"},{"value":"1861-2784","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,1]]},"assertion":[{"value":"6 February 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 August 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 August 2025","order":5,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":6,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The original version of this article was revised due to missing video file. Now it has been corrected.","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}}]}}