{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T15:13:28Z","timestamp":1774624408208,"version":"3.50.1"},"reference-count":90,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,10,4]],"date-time":"2022-10-04T00:00:00Z","timestamp":1664841600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Industrial robots and cobots are widely deployed in most industrial sectors. However, robotic programming still needs a lot of time and effort in small batch sizes, and it demands specific expertise and special training, especially when various robotic platforms are required. Actual low-code or no-code robotic programming solutions are exorbitant and meager. This work proposes a novel approach for no-code robotic programming for end-users with adequate or no expertise in industrial robotic. The proposed method ensures intuitive and fast robotic programming by utilizing a finite state machine with three layers of natural interactions based on hand gesture, finger gesture, and voice recognition. The implemented system combines intelligent computer vision and voice control capabilities. Using a vision system, the human could transfer spatial information of a 3D point, lines, and trajectories using hand and finger gestures. The voice recognition system will assist the user in parametrizing robot parameters and interacting with the robot\u2019s state machine. Furthermore, the proposed method will be validated and compared with state-of-the-art \u201cHand-Guiding\u201d cobot devices within real-world experiments. The results obtained are auspicious, and indicate the capability of this novel approach for real-world deployment in an industrial context.<\/jats:p>","DOI":"10.3389\/frobt.2022.1001955","type":"journal-article","created":{"date-parts":[[2022,10,4]],"date-time":"2022-10-04T07:57:13Z","timestamp":1664870233000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["No-code robotic programming for agile production: A new markerless-approach for multimodal natural interaction in a human-robot collaboration context"],"prefix":"10.3389","volume":"9","author":[{"given":"Jayanto","family":"Halim","sequence":"first","affiliation":[]},{"given":"Paul","family":"Eichler","sequence":"additional","affiliation":[]},{"given":"Sebastian","family":"Krusche","sequence":"additional","affiliation":[]},{"given":"Mohamad","family":"Bdiwi","sequence":"additional","affiliation":[]},{"given":"Steffen","family":"Ihlenfeldt","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2022,10,4]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"103845","DOI":"10.1016\/j.robot.2021.103845","article-title":"Designing user-centric programming aids for kinesthetic teaching of collaborative robots","volume":"145","author":"Ajaykumar","year":"2021","journal-title":"Robotics Aut. Syst."},{"key":"B2","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1007\/s00502-019-00741-4","article-title":"Skill-based programming of complex robotic assembly tasks for industrial application","volume":"136","author":"Akkaladevi","year":"2019","journal-title":"Elektrotech. Inftech."},{"key":"B3","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1016\/j.promfg.2020.10.022","article-title":"Programming by interactive demonstration for a human robot collaborative assembly","volume":"51","author":"Akkaladevi","year":"2020","journal-title":"Procedia Manuf."},{"key":"B4","first-page":"430","article-title":"On-line hand-eye calibration","author":"Andreff","year":"1999"},{"key":"B5","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/s12541-013-0306-5","article-title":"Prediction and compensation of relative position error along industrial robot end-effector paths","volume":"15","author":"Angelidis","year":"2014","journal-title":"Int. Suchy\u2019 J. Precis. Eng. Manuf."},{"key":"B6","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1016\/j.robot.2008.10.024","article-title":"A survey of robot learning from demonstration","volume":"57","author":"Argall","year":"2009","journal-title":"Robotics Aut. Syst."},{"key":"B7","first-page":"85","article-title":"Using finite state machines in introductory robotics","author":"Balogh","year":"2018"},{"key":"B8","first-page":"150","article-title":"Empirical study for measuring the mental states of humans during the interaction with heavy-duty industrial robots","author":"Bdiwi","year":"2021"},{"key":"B9","first-page":"40","article-title":"Automated assistance robot system for transferring model-free objects from\/to human hand using vision\/force control","author":"Bdiwi","year":""},{"key":"B10","first-page":"1","article-title":"Segmentation of model-free objects carried by human hand: Intended for human-robot interaction applications","author":"Bdiwi","year":""},{"key":"B11","first-page":"1","article-title":"Improved peg-in-hole (5-pin plug) task: Intended for charging electric vehicles by robot system automatically","author":"Bdiwi","year":"2015"},{"key":"B12","first-page":"1","article-title":"Handing-over model-free objects to human hand with the help of vision\/force robot control","author":"Bdiwi","year":""},{"key":"B13","doi-asserted-by":"publisher","first-page":"1280","DOI":"10.1016\/j.procir.2021.11.215","article-title":"Towards a cad-based automated robot offline-programming approach for disassembly","volume":"104","author":"Beck","year":"2021","journal-title":"Procedia CIRP"},{"key":"B14","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-540-30301-5_60","article-title":"Robot programming by demonstration","volume-title":"Springer handbook of robotics","author":"Billard","year":"2008"},{"key":"B15","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.procir.2018.02.028","article-title":"Intuitive robot programming using augmented reality","volume":"76","author":"Blankemeyer","year":"2018","journal-title":"Procedia CIRP"},{"key":"B16","first-page":"625","article-title":"Virtual reality for offline programming of robotic applications with online teaching methods","author":"Bolano","year":"2020"},{"key":"B17","doi-asserted-by":"publisher","first-page":"53901","DOI":"10.1109\/ACCESS.2020.2981378","article-title":"Hole-filling of realsense depth images using a color edge map","volume":"8","author":"Cho","year":"2020","journal-title":"IEEE Access"},{"key":"B18","first-page":"15","article-title":"Production systems flexibility: Theory and practice","author":"Chryssolouris","year":"2012"},{"key":"B19","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1177\/02783649922066213","article-title":"Hand-eye calibration using dual quaternions","volume":"18","author":"Daniilidis","year":"1999","journal-title":"Int. J. Robotics Res."},{"key":"B20","article-title":"Programming system for efficient use of industrial robots for deburring in sme environments","author":"Dietz","year":"2012"},{"key":"B21","volume-title":"DIN EN ISO 10218-1: Robotik sicherheitsanforderungen - teil 1: Industrieroboter","year":"2012"},{"key":"B22","volume-title":"Industrieroboter - sicherheitsanforderungen - Teil 2: Robotersysteme und Integration","year":"2021"},{"key":"B23","volume-title":"Din ISO\/TS 15066: Roboter und Robotikger\u00e4te - kollaborierende Roboter. Norm","year":"2017"},{"key":"B24","doi-asserted-by":"publisher","first-page":"172988141985166","DOI":"10.1177\/1729881419851665","article-title":"Stitching of depth and color images from multiple rgb-d sensors for extended field of view","volume":"16","author":"Ding","year":"2019","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B25","doi-asserted-by":"publisher","first-page":"2257","DOI":"10.1109\/tie.2014.2362095","article-title":"A markerless human\u2013robot interface using particle filter and kalman filter for dual robots","volume":"62","author":"Du","year":"2014","journal-title":"IEEE Trans. Ind. Electron."},{"key":"B26","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1016\/j.robot.2019.03.003","article-title":"Cobot programming for collaborative industrial tasks: An overview","volume":"116","author":"El Zaatari","year":"2019","journal-title":"Robotics Aut. Syst."},{"key":"B27","first-page":"993","article-title":"Learning generalizable surface cleaning actions from demonstration","author":"Elliott","year":"2017"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1177\/17298814211064767","article-title":"Novel surface optimization for trajectory reconstruction in industrial robot tasks","volume":"18","author":"Funes-Lora","year":"2021","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B29","doi-asserted-by":"publisher","first-page":"2280","DOI":"10.1016\/j.patcog.2014.01.005","article-title":"Automatic generation and detection of highly reliable fiducial markers under occlusion","volume":"47","author":"Garrido-Jurado","year":"2014","journal-title":"Pattern Recognit."},{"key":"B30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2010324.1964964","article-title":"Domain transform for edge-aware image and video processing","volume":"30","author":"Gastal","year":"2011","journal-title":"ACM Trans. Graph."},{"key":"B31","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-32552-1_54","volume-title":"Industrial robotics","author":"H\u00e4gele","year":"2016"},{"key":"B32","volume-title":"Fast robust peg-in-hole insertion with continuous visual servoing","author":"Haugaard","year":"2020"},{"key":"B33","doi-asserted-by":"publisher","first-page":"105676","DOI":"10.1016\/j.dib.2020.105676","article-title":"Hgm-4: A new multi-cameras dataset for hand gesture recognition","volume":"30","author":"Hoang","year":"2020","journal-title":"Data Brief"},{"key":"B34","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1177\/027836499501400301","article-title":"Hand-eye calibration","volume":"14","author":"Horaud","year":"1995","journal-title":"Int. J. Rob. Res."},{"key":"B35","first-page":"10","article-title":"Human-robot collaboration: a survey on the state of the art focusing on risk assessment","author":"Hornung","year":"2022"},{"key":"B36","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v34i07.6761","article-title":"Awr: Adaptive weighting regression for 3d hand pose estimation","author":"Huang","year":"2020"},{"key":"B37","first-page":"355","article-title":"Usability of force-based controllers in physical human-robot interaction","author":"Infante","year":"2011"},{"key":"B38","volume-title":"Intel RealSense D400 series product family","author":"Intel","year":"2015"},{"key":"B39","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/j.procir.2020.05.213","article-title":"Programming cobots by voice: A human-centered, web-based approach","volume":"97","author":"Ionescu","year":"2021","journal-title":"Procedia CIRP"},{"key":"B40","doi-asserted-by":"publisher","first-page":"767878","DOI":"10.3389\/frobt.2021.767878","article-title":"Quick setup of force-controlled industrial gluing tasks using learning from demonstration","volume":"8","author":"Iturrate","year":"2021","journal-title":"Front. Robot. AI"},{"key":"B41","first-page":"78","article-title":"Integration of vision\/force robot control for transporting different shaped\/colored objects from moving circular conveyor","author":"Jokesch","year":"2014"},{"key":"B42","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1016\/j.cirp.2022.04.074","article-title":"Robot polishing control with an active end effector based on macro-micro mechanism and the extended preston\u2019s law","volume":"71","author":"Kakinuma","year":"2022","journal-title":"CIRP Ann."},{"key":"B43","first-page":"781","article-title":"Experimental comparison of fiducial markers for pose estimation","author":"Kalaitzakis","year":"2020"},{"key":"B44","doi-asserted-by":"publisher","first-page":"102033","DOI":"10.1016\/j.rcim.2020.102033","article-title":"Human\u2013robot co-manipulation during surface tooling: A general framework based on impedance control, haptic rendering and discrete geometry","volume":"67","author":"Kana","year":"2021","journal-title":"Robotics Computer-Integrated Manuf."},{"key":"B45","doi-asserted-by":"publisher","first-page":"393","DOI":"10.14257\/ijsip.2016.9.4.34","article-title":"A review on automatic speech recognition architecture and approaches","volume":"9","author":"Karpagavalli","year":"2016","journal-title":"Int. J. Signal Process. Image Process. Pattern Recognit."},{"key":"B46","doi-asserted-by":"publisher","first-page":"254","DOI":"10.5772\/55640","article-title":"Spatial programming for industrial robots through task demonstration","volume":"10","author":"Lambrecht","year":"2013","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B47","first-page":"438","article-title":"Depth camera technology comparison and performance evaluation","author":"Langmann","year":"2012","journal-title":"ICPRAM"},{"key":"B48","volume-title":"A survey of robot learning from demonstrations for human-robot collaboration","author":"Lee","year":"2017"},{"key":"B49","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.jmapro.2020.03.047","article-title":"Automatic weld seam tracking of tube-to-tubesheet tig welding robot with multiple sensors","volume":"63","author":"Lei","year":"2021","journal-title":"J. Manuf. Process."},{"key":"B50","doi-asserted-by":"publisher","first-page":"172988141769556","DOI":"10.1177\/1729881417695560","article-title":"Real-time rgb-d image stitching using multiple kinects for improved field of view","volume":"14","author":"Li","year":"2017","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B51","first-page":"5354","article-title":"Eye-in-hand\/eye-to-hand multi-camera visual servoing","author":"Lippiello","year":"2005"},{"key":"B52","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1016\/j.cirpj.2020.11.015","article-title":"Sensorless haptic control for human-robot collaborative assembly","volume":"32","author":"Liu","year":"2021","journal-title":"CIRP J. Manuf. Sci. Technol."},{"key":"B53","doi-asserted-by":"publisher","first-page":"1188","DOI":"10.1016\/j.procir.2020.03.022","article-title":"Symbiotic human-robot collaboration: Multimodal control using function blocks","volume":"93","author":"Liu","year":"2020","journal-title":"Procedia CIRP"},{"key":"B54","first-page":"362","article-title":"Intel realsense sr305, d415 and l515: Experimental evaluation and comparison of depth estimation","author":"Louren\u00e7o","year":"2021"},{"key":"B55","doi-asserted-by":"publisher","first-page":"2633","DOI":"10.1109\/TVCG.2015.2513408","article-title":"Pose estimation for augmented reality: A hands-on survey","volume":"22","author":"Marchand","year":"2016","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"B56","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1108\/ir-11-2014-0413","article-title":"Manual guidance for industrial robot programming","volume":"42","author":"Massa","year":"2015","journal-title":"Industrial Robot An Int. J."},{"key":"B57","volume-title":"OpenMMLab pose estimation toolbox and benchmark","author":"MMPose-Contributors","year":"2020"},{"key":"B58","doi-asserted-by":"publisher","first-page":"896","DOI":"10.1016\/j.robot.2013.02.005","article-title":"Direct off-line robot programming via a common cad package","volume":"61","author":"Neto","year":"2013","journal-title":"Robotics Aut. Syst."},{"key":"B59","doi-asserted-by":"publisher","first-page":"717","DOI":"10.1109\/70.326576","article-title":"Robot sensor calibration: Solving ax=xb on the Euclidean group","volume":"10","author":"Park","year":"1994","journal-title":"IEEE Trans. Rob. Autom."},{"key":"B60","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1007\/s41315-018-0053-y","article-title":"Collaborative manipulation with multiple dual-arm robots under human guidance","volume":"2","author":"Peng","year":"2018","journal-title":"Int. J. Intell. Robot. Appl."},{"key":"B61","first-page":"517","article-title":"Camera calibration method based on opencv","volume-title":"Materials engineering and automatic control II (trans tech publications ltd)","author":"Qiao","year":"2013"},{"key":"B62","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1146\/annurev-control-100819-063206","article-title":"Recent advances in robot learning from demonstration","volume":"3","author":"Ravichandar","year":"2020","journal-title":"Annu. Rev. Control Robot. Auton. Syst."},{"key":"B63","volume-title":"User Manual UR10\/CB3 - original instructions (en)","author":"Robots","year":"2015"},{"key":"B64","volume-title":"Speech is 3x faster than typing for English and Mandarin text entry on mobile devices","author":"Ruan","year":"2016"},{"key":"B65","doi-asserted-by":"publisher","first-page":"1098","DOI":"10.1080\/0951192x.2019.1686177","article-title":"Operation of a haptic interface for offline programming of welding robots by applying a spring-damper model","volume":"32","author":"Sanchez-Diaz","year":"2019","journal-title":"Int. J. Comput. Integr. Manuf."},{"key":"B66","doi-asserted-by":"publisher","first-page":"7770","DOI":"10.3390\/s21227770","article-title":"Metrological characterization and comparison of d415, d455, l515 realsense devices in the close range","volume":"21","author":"Servi","year":"2021","journal-title":"Sensors"},{"key":"B67","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.494","article-title":"Hand keypoint detection in single images using multiview bootstrapping","author":"Simon","year":"2017"},{"key":"B68","doi-asserted-by":"publisher","first-page":"5976","DOI":"10.3390\/s21175976","article-title":"Programming robots by demonstration using augmented reality","volume":"21","author":"Soares","year":"2021","journal-title":"Sensors"},{"key":"B69","unstructured":"Mqtt version 3.1. 1\n            StandardO.\n          2014"},{"key":"B70","doi-asserted-by":"publisher","first-page":"923","DOI":"10.3390\/s22030923","article-title":"Robot system assistant (rosa): Towards intuitive multi-modal and multi-device human-robot interaction","volume":"22","author":"Strazdas","year":"2022","journal-title":"Sensors"},{"key":"B71","first-page":"6825","article-title":"Onepose: One-shot object pose estimation without cad models","author":"Sun","year":"2022"},{"key":"B72","doi-asserted-by":"publisher","first-page":"1467","DOI":"10.3390\/app11041467","article-title":"Painting path planning for a painting robot with a realsense depth sensor","volume":"11","author":"Tadic","year":"2021","journal-title":"Appl. Sci."},{"key":"B73","doi-asserted-by":"publisher","first-page":"639","DOI":"10.1007\/s00170-016-8527-2","article-title":"Polishing pressure investigations of robot automatic polishing on curved surfaces","volume":"87","author":"Tian","year":"2016","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"B74","first-page":"1","article-title":"User-friendly programming of flexible assembly applications with collaborative robots","author":"Tirmizi","year":"2019"},{"key":"B75","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1109\/70.34770","article-title":"A new technique for fully autonomous and efficient 3d robotics hand\/eye calibration","volume":"5","author":"Tsai","year":"1989","journal-title":"IEEE Trans. Rob. Autom."},{"key":"B76","first-page":"205","article-title":"Incrementally assisted kinesthetic teaching for programming by demonstration","author":"Tykal","year":"2016"},{"key":"B77","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1108\/01439911211268796","article-title":"Pick-and-place application development using voice and visual commands","volume":"39","author":"van Delden","year":"2012","journal-title":"Industrial Robot An Int. J."},{"key":"B78","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1016\/j.mechatronics.2018.02.009","article-title":"Survey on human\u2013robot collaboration in industrial settings: Safety, intuitive interfaces and applications","volume":"55","author":"Villani","year":"2018","journal-title":"Mechatronics"},{"key":"B79","volume-title":"vicCONTROL industrial version 6.3.0 - User guide - phytec Voice Control Kits (phyBOARD\u00ae-Mira). voice INTER connect GmbH","year":"2022"},{"key":"B80","doi-asserted-by":"publisher","first-page":"56","DOI":"10.5898\/jhri.2.1.wrede","article-title":"A user study on kinesthetic teaching of redundant robots in task and configuration space","volume":"2","author":"Wrede","year":"2013","journal-title":"J. Hum. Robot. Interact."},{"key":"B81","volume-title":"Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes","author":"Xiang","year":"2017"},{"key":"B82","doi-asserted-by":"publisher","first-page":"1512","DOI":"10.1109\/tase.2018.2874454","article-title":"Haptics electromyography perception and learning enhanced intelligence for teleoperated robot","volume":"16","author":"Yang","year":"","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"B83","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1109\/jsen.2018.2877976","article-title":"A novel 3-d path extraction method for arc welding robot based on stereo structured light sensor","volume":"19","author":"Yang","year":"","journal-title":"IEEE Sens. J."},{"key":"B84","first-page":"1","article-title":"Virtual experiments on aruco and apriltag systems comparison for fiducial marker rotation resistance under noisy sensory data","author":"Zakiev","year":"2020"},{"key":"B85","doi-asserted-by":"publisher","first-page":"101923","DOI":"10.1016\/j.rcim.2019.101923","article-title":"Accurate dynamic modeling and control parameters design of an industrial hybrid spray-painting robot","volume":"63","author":"Zhang","year":"","journal-title":"Robotics Computer-Integrated Manuf."},{"key":"B86","volume-title":"Mediapipe hands: On-device real-time hand tracking","author":"Zhang","year":""},{"key":"B87","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1007\/s40436-020-00303-4","article-title":"Robot programming by demonstration: A novel system for robot trajectory programming based on robot operating system","volume":"8","author":"Zhang","year":"","journal-title":"Adv. Manuf."},{"key":"B88","doi-asserted-by":"publisher","first-page":"5204","DOI":"10.1109\/tii.2019.2900119","article-title":"A sensorless hand guiding scheme based on model identification and control for industrial robot","volume":"15","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Ind. Inf."},{"key":"B89","doi-asserted-by":"publisher","first-page":"168781402110040","DOI":"10.1177\/16878140211004034","article-title":"A hybrid control strategy for grinding and polishing robot based on adaptive impedance control","volume":"13","author":"Zhou","year":"2021","journal-title":"Adv. Mech. Eng."},{"key":"B90","volume-title":"Modularization of research software for collaborative open source development","author":"Zirkelbach","year":"2019"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2022.1001955\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,7]],"date-time":"2022-10-07T02:36:26Z","timestamp":1665110186000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2022.1001955\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,4]]},"references-count":90,"alternative-id":["10.3389\/frobt.2022.1001955"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2022.1001955","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,4]]},"article-number":"1001955"}}