{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T20:03:13Z","timestamp":1784664193052,"version":"3.55.0"},"reference-count":27,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T00:00:00Z","timestamp":1648684800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004663","name":"Ministry of Science and Technology, Taiwan","doi-asserted-by":"publisher","award":["MOST 110-2634-F-003 -006 and MOST 110-2634-F-003 -007"],"award-info":[{"award-number":["MOST 110-2634-F-003 -006 and MOST 110-2634-F-003 -007"]}],"id":[{"id":"10.13039\/501100004663","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Robotic arms have been widely used in various industries and have the advantages of cost savings, high productivity, and efficiency. Although robotic arms are good at increasing efficiency in repetitive tasks, they still need to be re-programmed and optimized when new tasks are to be deployed, resulting in detrimental downtime and high cost. It is therefore the objective of this paper to present a learning from demonstration (LfD) robotic system to provide a more intuitive way for robots to efficiently perform tasks through learning from human demonstration on the basis of two major components: understanding through human demonstration and reproduction by robot arm. To understand human demonstration, we propose a vision-based spatial-temporal action detection method to detect human actions that focuses on meticulous hand movement in real time to establish an action base. An object trajectory inductive method is then proposed to obtain a key path for objects manipulated by the human through multiple demonstrations. In robot reproduction, we integrate the sequence of actions in the action base and the key path derived by the object trajectory inductive method for motion planning to reproduce the task demonstrated by the human user. Because of the capability of learning from demonstration, the robot can reproduce the tasks that the human demonstrated with the help of vision sensors in unseen contexts.<\/jats:p>","DOI":"10.3390\/s22072678","type":"journal-article","created":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T21:34:29Z","timestamp":1648762469000},"page":"2678","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Vision-Based Learning from Demonstration System for Robot Arms"],"prefix":"10.3390","volume":"22","author":[{"given":"Pin-Jui","family":"Hwang","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, National Taiwan Normal University, Taipei 106, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3697-8401","authenticated-orcid":false,"given":"Chen-Chien","family":"Hsu","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, National Taiwan Normal University, Taipei 106, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Po-Yung","family":"Chou","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, National Taiwan Normal University, Taipei 106, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei-Yen","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, National Taiwan Normal University, Taipei 106, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheng-Hung","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, National Taiwan Normal University, Taipei 106, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1111\/poms.12797","article-title":"Throughput Optimization in Circular Dual-Gripper Robotic Cells","volume":"27","author":"Jung","year":"2018","journal-title":"Prod. Oper. Manag."},{"key":"ref_2","first-page":"1","article-title":"A Scalarization-Based Method for Multiple Part-Type Scheduling of Two-Machine Robotic Systems with Non-Destructive Testing Technologies","volume":"10","author":"Foumani","year":"2019","journal-title":"Iran. J. Oper. Res."},{"key":"ref_3","unstructured":"Lieberman, H. (2001). Your Wish Is My Command: Programming by Example, Morgan Kaufmann."},{"key":"ref_4","unstructured":"Kurlander, D., Cypher, A., and Halbert, D.C. (1993). Watch What I Do: Programming by Demonstration, MIT Press."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Billard, A., Calinon, S., Dillmann, R., and Schaal, S. (2008). Survey: Robot Programming by Demonstration, Springer.","DOI":"10.1007\/978-3-540-30301-5_60"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Reiner, B., Ertel, W., Posenauer, H., and Schneider, M. (2014, January 14\u201318). Lat: A Simple Learning from Demonstration Method. Proceedings of the 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA.","DOI":"10.1109\/IROS.2014.6943190"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1109\/TSMCB.2006.886951","article-title":"Incremental Learning of Tasks from User Demonstrations, Past Experiences, and Vocal Comments","volume":"37","author":"Pardowitz","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. Part B"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Calinon, S., and Billard, A. (2007, January 26\u201329). Active Teaching in Robot Programming by Demonstration. Proceedings of the RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication, Jeju, Korea.","DOI":"10.1109\/ROMAN.2007.4415177"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Radensky, M., Li, T.J.-J., and Myers, B.A. (2018, January 1\u20134). How End Users Express Conditionals in Programming by Demonstration for Mobile Apps. Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC), Lisbon, Portugal.","DOI":"10.1109\/VLHCC.2018.8506492"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1016\/j.robot.2008.10.024","article-title":"A Survey of Robot Learning from Demonstration","volume":"57","author":"Argall","year":"2009","journal-title":"Robot. Auton. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1109\/MCE.2019.2956202","article-title":"Development of a Mimic Robot\u2014Learning From Demonstration Incorporating Object Detection and Multiaction Recognition","volume":"9","author":"Hwang","year":"2020","journal-title":"IEEE Consum. Electron. Mag."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Hwang, P.-J., Hsu, C.-C., and Wang, W.-Y. (2019, January 19\u201321). Development of A Mimic Robot: Learning from Human Demonstration to Manipulate A Coffee Maker as An Example. Proceedings of the 2019 IEEE 23rd International Symposium on Consumer Technologies (ISCT), Ancona, Italy.","DOI":"10.1109\/ISCE.2019.8901025"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1100","DOI":"10.1109\/TSMC.2020.3013904","article-title":"Learning-Based Kinematic Control Using Position and Velocity Errors for Robot Trajectory Tracking","volume":"52","author":"Xu","year":"2020","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Lentini, G., Grioli, G., Catalano, M.G., and Bicchi, A. (August, January 31). Robot Programming Without Coding. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9196904"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Cao, Z., Hu, H., Zhao, Z., and Lou, Y. (2019, January 6\u20138). Robot Programming by Demonstration with Local Human Correction for Assembly. Proceedings of the 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dali, China.","DOI":"10.1109\/ROBIO49542.2019.8961854"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Papageorgiou, D., and Doulgeri, Z. (September, January 31). Learning by Demonstration for Constrained Tasks. Proceedings of the 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Naples, Italy.","DOI":"10.1109\/RO-MAN47096.2020.9223579"},{"key":"ref_17","unstructured":"Hwang, P.-J., Hsu, C.-C., Wang, W.-Y., and Chiang, H.-H. (2020, January 4\u20136). Robot Learning from Demonstration based on Action and Object Recognition. Proceedings of the IEEE International Conference on Consumer Electronics, Las Vegas, NV, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Tsai, J.-K., Hsu, C.-C., Wang, W.-Y., and Huang, S.-K. (2020). Deep Learning-Based Real-Time Multiple-Person Action Recognition System. Sensors, 20.","DOI":"10.3390\/s20174758"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2017, January 21\u201326). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Feichtenhofer, C., Pinz, A., and Zisserman, A. (2016, January 27\u201330). Convolutional Two-Stream Network Fusion for Video Action Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.213"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Habib, S., Hussain, A., Albattah, W., Islam, M., Khan, S., Khan, R.U., and Khan, K. (2021). Abnormal Activity Recognition from Surveillance Videos Using Convolutional Neural Network. Sensors, 21.","DOI":"10.3390\/s21248291"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1007\/s12369-012-0160-0","article-title":"Keyframe-Based Learning from Demonstration","volume":"4","author":"Akgun","year":"2012","journal-title":"Int. J. Soc. Robot."},{"key":"ref_23","unstructured":"Muench, S., Kreuziger, J., Kaiser, M., and Dillman, R. (1994, January 25\u201327). Robot Programming by Demonstration (rpd)-Using Machine Learning and User Interaction Methods for the Development of Easy and Comfortable Robot Programming Systems. Proceedings of the International Symposium on Industrial Robots (ISIR \u201994), Hannover, Germany."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1109\/70.326567","article-title":"Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics","volume":"10","author":"Yang","year":"1994","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_25","unstructured":"Hovland, G.E., Sikka, P., and McCarragher, B.J. (1996, January 22\u201328). Skill Acquisition from Human Demonstration Using a Hidden Markov Model. Proceedings of the IEEE International Conference on Robotics and Automation, Minneapolis, MN, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kim, M.G., Lee, S.H., and Suh, I.H. (2014, January 15\u201318). Learning of Social Skills for Human-Robot Interaction by Hierarchical Hmm and Interaction Dynamics. Proceedings of the 2014 International Conference on Electronics, Information and Communications (ICEIC), Kota Kinabalu, Malaysia.","DOI":"10.1109\/ELINFOCOM.2014.6914380"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017, January 22\u201329). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.324"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/7\/2678\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:47:10Z","timestamp":1760136430000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/7\/2678"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,31]]},"references-count":27,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["s22072678"],"URL":"https:\/\/doi.org\/10.3390\/s22072678","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,31]]}}}