{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T15:41:59Z","timestamp":1775144519573,"version":"3.50.1"},"reference-count":25,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T00:00:00Z","timestamp":1678924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100007826","name":"Technische Universit\u00e4t Ilmenau","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100007826","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>The concept of Industry 4.0 brings the change of industry manufacturing patterns that become more efficient and more flexible. In response to this tendency, an efficient robot teaching approach without complex programming has become a popular research direction. Therefore, we propose an interactive finger-touch based robot teaching schema using a multimodal 3D image (color (RGB), thermal (T) and point cloud (3D)) processing. Here, the resulting heat trace touching the object surface will be analyzed on multimodal data, in order to precisely identify the true hand\/object contact points. These identified contact points are used to calculate the robot path directly. To optimize the identification of the contact points we propose a calculation scheme using a number of anchor points which are first predicted by hand\/object point cloud segmentation. Subsequently a probability density function is defined to calculate the prior probability distribution of true finger trace. The temperature in the neighborhood of each anchor point is then dynamically analyzed to calculate the likelihood. Experiments show that the trajectories estimated by our multimodal method have significantly better accuracy and smoothness than only by analyzing point cloud and static temperature distribution.<\/jats:p>","DOI":"10.3389\/frobt.2023.1120357","type":"journal-article","created":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T04:45:21Z","timestamp":1678941921000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Interactive robot teaching based on finger trajectory using multimodal RGB-D-T-data"],"prefix":"10.3389","volume":"10","author":[{"given":"Yan","family":"Zhang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard","family":"F\u00fctterer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gunther","family":"Notni","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,3,16]]},"reference":[{"key":"B1","first-page":"2210","article-title":"Augmented reality based teaching pendant for industrial robot","volume-title":"2012 12th international conference on control, automation and systems","author":"Abbas","year":"2012"},{"key":"B2","first-page":"8330","article-title":"Augmented skeleton space transfer for depth-based hand pose estimation","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Baek","year":"2018"},{"key":"B3","first-page":"1","article-title":"Finger pointer based human machine interaction for selected quality checks of industrial work pieces","volume-title":"ISR 2020; 52th international symposium on Robotics (VDE)","author":"Braeuer-Burchardt","year":"2020"},{"key":"B4","first-page":"11260","article-title":"Handfoldingnet: A 3d hand pose estimation network using multiscale-feature guided folding of a 2d hand skeleton","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"Cheng","year":"2021"},{"key":"B5","doi-asserted-by":"publisher","first-page":"9571","DOI":"10.1109\/tie.2018.2823667","article-title":"Online robot teaching with natural human\u2013robot interaction","volume":"65","author":"Du","year":"2018","journal-title":"IEEE Trans. Industrial Electron."},{"key":"B6","unstructured":"Flir a35 product overview2022"},{"key":"B7","unstructured":"Genie nano c1280 product overview2022"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1001955","DOI":"10.3389\/frobt.2022.1001955","article-title":"No-Code robotic programming for agile production: A new markerless-approach for multimodal natural interaction in a human-robot collaboration context","volume":"9","author":"Halim","year":"2022","journal-title":"Front. Robotics AI"},{"key":"B9","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1038\/s41377-018-0072-3","article-title":"Gobo projection for 3d measurements at highest frame rates: A performance analysis","volume":"7","author":"Heist","year":"2018","journal-title":"Light: Sci. Appl."},{"key":"B10","first-page":"11108","article-title":"Randla-net: Efficient semantic segmentation of large-scale point clouds","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Hu","year":"2020"},{"key":"B11","first-page":"314","article-title":"Vr-based robot programming and simulation system for an industrial robot","volume":"15","author":"Jen","year":"2008","journal-title":"Int. J. Industrial Eng."},{"key":"B12","doi-asserted-by":"publisher","first-page":"453","DOI":"10.3390\/s16040453","article-title":"Human detection based on the generation of a background image and fuzzy system by using a thermal camera","volume":"16","author":"Jeon","year":"2016","journal-title":"Sensors"},{"key":"B13","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1515\/pjbr-2022-0004","article-title":"Study of industrial interactive design system based on virtual reality teaching technology in industrial robot","volume":"13","author":"Liu","year":"2022","journal-title":"Paladyn, J. Behav. Robotics"},{"key":"B14","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1007\/s12008-018-0516-2","article-title":"Off-line programming of an industrial robot in a virtual reality environment","volume":"13","author":"Manou","year":"2019","journal-title":"Int. J. Interact. Des. Manuf. (IJIDeM)"},{"key":"B15","volume-title":"Real-time 2d multi-person pose estimation on cpu: Lightweight openpose","author":"Osokin","year":"2018"},{"key":"B16","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1109\/ISMAR.2003.1240739","article-title":"Augmented reality for programming industrial robots","volume-title":"The second IEEE and ACM international symposium on mixed and augmented reality, 2003. Proceedings","author":"Pettersen","year":"2003"},{"key":"B17","doi-asserted-by":"publisher","first-page":"103446","DOI":"10.1016\/j.compind.2021.103446","article-title":"Towards the adoption of virtual reality training systems for the self-tuition of industrial robot operators: A case study at kuka","volume":"129","author":"Prattic\u00f2","year":"2021","journal-title":"Comput. Industry"},{"key":"B18","first-page":"652","article-title":"Pointnet: Deep learning on point sets for 3d classification and segmentation","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"Qi","year":""},{"key":"B19","article-title":"Pointnet++: Deep hierarchical feature learning on point sets in a metric space","volume":"30","author":"Qi","year":"","journal-title":"Adv. neural Inf. Process. Syst."},{"key":"B20","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/ROMAN.2016.7745108","article-title":"Augmented reality for industrial robot programmers: Workload analysis for task-based, augmented reality-supported robot control","volume-title":"2016 25th IEEE international symposium on robot and human interactive communication (RO-MAN)","author":"Stadler","year":"2016"},{"key":"B21","doi-asserted-by":"crossref","first-page":"1809","DOI":"10.1109\/SMC.2018.00313","article-title":"Development of a 3d ar-based interface for industrial robot manipulators","volume-title":"2018 IEEE international conference on systems, man, and cybernetics (SMC)","author":"Su","year":"2018"},{"key":"B22","unstructured":"Wandelbots teaching product overview2022"},{"key":"B23","doi-asserted-by":"publisher","first-page":"e109692","DOI":"10.1371\/journal.pone.0109692","article-title":"Virtual reality based support system for layout planning and programming of an industrial robotic work cell","volume":"9","author":"Yap","year":"2014","journal-title":"PloS one"},{"key":"B24","volume-title":"Mediapipe hands: On-device real-time hand tracking","author":"Zhang","year":"2020"},{"key":"B25","doi-asserted-by":"publisher","first-page":"5676","DOI":"10.3390\/s21165676","article-title":"Point cloud hand\u2013object segmentation using multimodal imaging with thermal and color data for safe robotic object handover","volume":"21","author":"Zhang","year":"2021","journal-title":"Sensors"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1120357\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T04:45:37Z","timestamp":1678941937000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1120357\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,16]]},"references-count":25,"alternative-id":["10.3389\/frobt.2023.1120357"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1120357","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,16]]},"article-number":"1120357"}}