{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:34:38Z","timestamp":1791567278520,"version":"4.3.4"},"reference-count":34,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2019,10,22]],"date-time":"2019-10-22T00:00:00Z","timestamp":1571702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this paper, the application of Augmented Reality (AR) for the control and adjustment of robots has been developed, with the aim of making interaction and adjustment of robots easier and more accurate from a remote location. A LeapMotion sensor based controller has been investigated to track the movement of the operator hands. The data from the controller allows gestures and the position of the hand palm\u2019s central point to be detected and tracked. A Kinect V2 camera is able to measure the corresponding motion velocities in x, y, z directions after our investigated post-processing algorithm is fulfilled. Unreal Engine 4 is used to create an AR environment for the user to monitor the control process immersively. Kalman filtering (KF) algorithm is employed to fuse the position signals from the LeapMotion sensor with the velocity signals from the Kinect camera sensor, respectively. The fused\/optimal data are sent to teleoperate a Baxter robot in real-time by User Datagram Protocol (UDP). Several experiments have been conducted to test the validation of the proposed method.<\/jats:p>","DOI":"10.3390\/s19204586","type":"journal-article","created":{"date-parts":[[2019,10,23]],"date-time":"2019-10-23T11:46:59Z","timestamp":1571831219000},"page":"4586","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7851-0260","authenticated-orcid":false,"given":"Chunxu","family":"Li","sequence":"first","affiliation":[{"name":"ASTUTE 2020 (Advanced Sustainable Manufacturing Technologies), Swansea University, Swansea SA1 8EN, UK"},{"name":"School of Automation and Electrical Engineering, Qingdao University of Science and Technology, Qingdao 266042, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashraf","family":"Fahmy","sequence":"additional","affiliation":[{"name":"ASTUTE 2020 (Advanced Sustainable Manufacturing Technologies), Swansea University, Swansea SA1 8EN, UK"},{"name":"Department of Electrical Power and Machines, Helwan University, Helwan 11795, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johann","family":"Sienz","sequence":"additional","affiliation":[{"name":"ASTUTE 2020 (Advanced Sustainable Manufacturing Technologies), Swansea University, Swansea SA1 8EN, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1561\/1100000005","article-title":"Human\u2013robot interaction: A survey","volume":"1","author":"Goodrich","year":"2008","journal-title":"Found. Trends\u00ae Hum. Comput. Interact."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bartneck, C., and Forlizzi, J. (2004). Shaping human-robot interaction: Understanding the social aspects of intelligent robotic products. CHI\u201904 Extended Abstracts on Human Factors in Computing Systems, ACM.","DOI":"10.1145\/985921.986205"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bremner, P., and Leonards, U. (2015, January 26\u201330). Efficiency of speech and iconic gesture integration for robotic and human communicators-a direct comparison. Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA.","DOI":"10.1109\/ICRA.2015.7139460"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1163\/156855307780131992","article-title":"Reactive robot system using a haptic interface: An active interaction to transfer skills from the robot to unskilled persons","volume":"21","author":"Solis","year":"2007","journal-title":"Adv. Robot."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1109\/TASE.2017.2743000","article-title":"Interface Design of a Physical Human\u2013Robot Interaction System for Human Impedance Adaptive Skill Transfer","volume":"15","author":"Yang","year":"2018","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Pastor, P., Hoffmann, H., Asfour, T., and Schaal, S. (2009, January 12\u201317). Learning and generalization of motor skills by learning from demonstration. Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan.","DOI":"10.1109\/ROBOT.2009.5152385"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Faion, F., Friedberger, S., Zea, A., and Hanebeck, U.D. (2012, January 7\u201312). Intelligent sensor-scheduling for multi-kinect-tracking. Proceedings of the 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.","DOI":"10.1109\/IROS.2012.6386007"},{"key":"ref_8","unstructured":"Nechyba, M.C., and Xu, Y. (1995, January 5\u20139). Human skill transfer: Neural networks as learners and teachers. Proceedings of the 1995 IEEE\/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots, Pittsburgh, PA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1049\/ccs.2018.0005","article-title":"Segmentation and generalization for writing skills transfer from humans to robots","volume":"1","author":"Li","year":"2019","journal-title":"Cognit. Comput. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Peppoloni, L., Brizzi, F., Ruffaldi, E., and Avizzano, C.A. (2015, January 13\u201315). Augmented reality-aided tele-presence system for robot manipulation in industrial manufacturing. Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology, Beijing, China.","DOI":"10.1145\/2821592.2821620"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1552","DOI":"10.1016\/j.promfg.2017.07.292","article-title":"A Stereo-Panoramic Telepresence System for Construction Machines","volume":"11","author":"Tripicchio","year":"2017","journal-title":"Procedia Manuf."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2072","DOI":"10.1109\/LRA.2017.2720850","article-title":"A novel wearable haptic controller for teleoperating robotic platforms","volume":"2","author":"Satler","year":"2017","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Quintero, C.P., Li, S., Pan, M.K., Chan, W.P., Van der Loos, H.M., and Croft, E. (2018, January 1\u20135). Robot programming through augmented trajectories in augmented reality. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593700"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Lee, D., and Park, Y.S. (2018, January 1\u20135). Implementation of Augmented Teleoperation System Based on Robot Operating System (ROS). Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594482"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"011006","DOI":"10.1115\/1.4025810","article-title":"Toward safe human robot collaboration by using multiple kinects based real-time human tracking","volume":"14","author":"Morato","year":"2014","journal-title":"J. Comput. Inf. Sci. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.mechatronics.2017.12.009","article-title":"Trajectory generation algorithm for safe human-robot collaboration based on multiple depth sensor measurements","volume":"55","author":"Ragaglia","year":"2018","journal-title":"Mechatronics"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Casalino, A., Guzman, S., Zanchettin, A.M., and Rocco, P. (2018, January 1\u20135). Human pose estimation in presence of occlusion using depth camera sensors, in human-robot coexistence scenarios. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593816"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.inffus.2011.09.004","article-title":"Distributed weighted robust Kalman filter fusion for uncertain systems with autocorrelated and cross-correlated noises","volume":"14","author":"Feng","year":"2013","journal-title":"Inf. Fusion"},{"key":"ref_19","unstructured":"Jenny, S. (2017). Enhancing Tourism with Augmented and Virtual Reality, HAMK."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1117\/12.197321","article-title":"Augmented reality: A class of displays on the reality-virtuality continuum","volume":"Volume 2351","author":"Milgram","year":"1995","journal-title":"Telemanipulator and Telepresence Technologies"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1186\/s40327-017-0058-x","article-title":"Integrating building information modeling and virtual reality development engines for building indoor lighting design","volume":"5","author":"Natephra","year":"2017","journal-title":"Vis. Eng."},{"key":"ref_22","unstructured":"Dang, T.L. (2017). Level Designing in Game Engine, Helsinki Metropolia University of Applied Sciences."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Sanders, A. (2016). An Introduction to Unreal Engine 4, AK Peters\/CRC Press.","DOI":"10.1201\/9781315382555"},{"key":"ref_24","unstructured":"Tran, N., Rands, J., and Williams, T. (2018, January 5\u20138). A Hands-Free Virtual-Reality Teleoperation Interface for Wizard-of-Oz Control. Proceedings of the 1st International Workshop on Virtual, Augmented, and Mixed Reality for HRI, Chicago, IL, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Marin, G., Dominio, F., and Zanuttigh, P. (2014, January 27\u201330). Hand gesture recognition with leap motion and kinect devices. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7025313"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1188","DOI":"10.1109\/LSP.2016.2590470","article-title":"Dynamic hand gesture recognition with leap motion controller","volume":"23","author":"Lu","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_27","unstructured":"Amon, C., Fuhrmann, F., and Graf, F. (2014, January 16\u201317). Evaluation of the spatial resolution accuracy of the face tracking system for kinect for windows v1 and v2. Proceedings of the 6th Congress of the Alps Adria Acoustics Association, Graz, Austria."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Li, C., Yang, C., Liang, P., Cangelosi, A., and Wan, J. (2016, January 18\u201320). Development of kinect based teleoperation of nao robot. Proceedings of the 2016 International Conference on Advanced Robotics and Mechatronics (ICARM), Macau, China.","DOI":"10.1109\/ICARM.2016.7606908"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1750023","DOI":"10.1142\/S0219843617500232","article-title":"Visual servoing of humanoid dual-arm robot with neural learning enhanced skill transferring control","volume":"15","author":"Yang","year":"2018","journal-title":"Int. J. Hum. Robot."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1109\/70.313105","article-title":"Simultaneous robot\/world and tool\/flange calibration by solving homogeneous transformation equations of the form AX= YB","volume":"10","author":"Zhuang","year":"1994","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1186\/1475-925X-13-88","article-title":"Arm movement speed assessment via a Kinect camera: A preliminary study in healthy subjects","volume":"13","author":"Elgendi","year":"2014","journal-title":"Biomed. Eng. Online"},{"key":"ref_32","first-page":"1","article-title":"Multirate Adaptive Kalman Filter for Marine Integrated Navigation System","volume":"70","author":"Davari","year":"2016","journal-title":"J. Navig."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1221","DOI":"10.3182\/20120711-3-BE-2027.00188","article-title":"On continuous-discrete cubature Kalman filtering","volume":"45","author":"Solin","year":"2012","journal-title":"IFAC Proc. Vol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1080\/21642583.2017.1300109","article-title":"Teleoperation control of Baxter robot using Kalman filter-based sensor fusion","volume":"5","author":"Li","year":"2017","journal-title":"Syst. Sci. Control Eng."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/20\/4586\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:28:20Z","timestamp":1760189300000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/20\/4586"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,22]]},"references-count":34,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["s19204586"],"URL":"https:\/\/doi.org\/10.3390\/s19204586","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,22]]}}}