{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T06:42:43Z","timestamp":1774161763919,"version":"3.50.1"},"reference-count":51,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2018,2,1]],"date-time":"2018-02-01T00:00:00Z","timestamp":1517443200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Teleoperated mobile robots, equipped with object manipulation capabilities, provide safe means for executing dangerous tasks in hazardous environments without putting humans at risk. However, mainly due to a communication delay, complex operator interfaces and insufficient Situational Awareness (SA), the task productivity of telerobots remains inferior to human workers. This paper addresses the shortcomings of telerobots by proposing a combined approach of (i) a scalable and intuitive operator interface with gestural and verbal input, (ii) improved Situational Awareness (SA) through sensor fusion according to documented best practices, (iii) integrated virtual fixtures for task simplification and minimizing the operator\u2019s cognitive burden and (iv) integrated semiautonomous behaviors that further reduce cognitive burden and negate the impact of communication delays, execution latency and\/or failures. The proposed teleoperation system, TeMoto, is implemented using ROS (Robot Operating System) to ensure hardware agnosticism, extensibility and community access. The operator\u2019s command interface consists of a Leap Motion Controller for hand tracking, Griffin PowerMate USB as turn knob for scaling and a microphone for speech input. TeMoto is evaluated on multiple robots including two mobile manipulator platforms. In addition to standard, task-specific evaluation techniques (completion time, user studies, number of steps, etc.)\u2014which are platform and task dependent and thus difficult to scale\u2014this paper presents additional metrics for evaluating the user interface including task-independent criteria for measuring generalized (i) task completion efficiency and (ii) operator context switching.<\/jats:p>","DOI":"10.3390\/robotics7010009","type":"journal-article","created":{"date-parts":[[2018,2,2]],"date-time":"2018-02-02T04:20:50Z","timestamp":1517545250000},"page":"9","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["TeMoto: Intuitive Multi-Range Telerobotic System with Natural Gestural and Verbal Instruction Interface"],"prefix":"10.3390","volume":"7","author":[{"given":"Robert","family":"Valner","sequence":"first","affiliation":[{"name":"Intelligent Materials and Systems Lab, University of Tartu, Nooruse 1, 50411 Tartu, Estonia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1720-1509","authenticated-orcid":false,"given":"Karl","family":"Kruusam\u00e4e","sequence":"additional","affiliation":[{"name":"Intelligent Materials and Systems Lab, University of Tartu, Nooruse 1, 50411 Tartu, Estonia"},{"name":"Nuclear and Applied Robotics Group, University of Texas at Austin, 10100 Burnet Rd, Austin, TX 78758, USA"}]},{"given":"Mitch","family":"Pryor","sequence":"additional","affiliation":[{"name":"Nuclear and Applied Robotics Group, University of Texas at Austin, 10100 Burnet Rd, Austin, TX 78758, USA"}]}],"member":"1968","published-online":{"date-parts":[[2018,2,1]]},"reference":[{"key":"ref_1","unstructured":"De Gregory, J., Lujan, R., Haley, D., and Hamel, W. (2001, January 1). Robotics and intelligent machines: A DOE critical technology roadmap. Proceedings of the 2001 Waste Management Symposium, Tucson, AZ, USA."},{"key":"ref_2","unstructured":"Rimando, R. (2017, October 22). EM Science of Safety (SoS): Robotics Technology Thrust, Available online: https:\/\/energy.gov\/sites\/prod\/files\/2016\/09\/f33\/DOE%20NCW%20-%20160915%20FINAL%20REV01.pdf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1002\/rob.21439","article-title":"Emergency response to the nuclear accident at the Fukushima Daiichi Nuclear Power Plants using mobile rescue robots","volume":"30","author":"Nagatani","year":"2013","journal-title":"J. Field Robot."},{"key":"ref_4","unstructured":"Asada, H. (2017, January 26). The Current Status and Future Strategic Plan of Decommissioning of Fukushima Daiichi Nuclear Power Stations Using Remote-Controlled Robot Technology. Proceedings of the IROS Open Forum on the Future of Robotics and Automation in Nuclear Facilities & Environments, Vancouver, CA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1238937","DOI":"10.1126\/science.1238937","article-title":"Volatile, Isotope and Organic Analysis of Martian Fines with the Mars Curiosity Rover","volume":"341","author":"Leshin","year":"2013","journal-title":"Science"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1177\/1557234X13510679","article-title":"Space Telerobotics: Unique Challenges to Human-Robot Collaboration in Space","volume":"9","author":"Fong","year":"2013","journal-title":"Rev. Hum. Factors Ergon."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4103\/0972-9941.51313","article-title":"Robotic assisted minimally invasive surgery","volume":"5","author":"Palep","year":"2009","journal-title":"J. Minimal Access Surg."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3140","DOI":"10.1021\/ie9905815","article-title":"Modification of Targets and Processes for Conversion of 99Mo Production from High- to Low-Enriched Uranium","volume":"39","author":"Vandegrift","year":"2000","journal-title":"Ind. Eng. Chem. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1109\/3468.844354","article-title":"A model for types and levels of human interaction with automation","volume":"30","author":"Parasuraman","year":"2000","journal-title":"IEEE Trans. Syst. Man Cybern. Part Syst. Hum."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1109\/TSMCA.2005.850599","article-title":"Shared Understanding for Collaborative Control","volume":"35","author":"Bruemmer","year":"2005","journal-title":"IEEE Trans. Syst. Man Cybern. Part Syst. Hum."},{"key":"ref_11","unstructured":"Brabec, C., Schroeder, K., Williams, J., O\u2019Neil, B., and Pryor, M. (2011, January 7\u201310). Reducing the operator\u2019s burden during teleoperation involving contact tasks. Proceedings of the 2011 ANS International Topical Meeting on Robotics and Remote Systems, Knoxville, TN, USA."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1109\/70.258052","article-title":"Space Teleoperation through Time Delay: Review and Prognosis","volume":"9","author":"Sheridan","year":"1993","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_13","unstructured":"Thompson, J., and Pryor, M. (2014, January 14). Spatial interface for user-centric robotic teleoperation and demonstration on a high precision task. Proceedings of the Workshop on Tele-operation at the IEEE Conference on Intelligent Robots and Systems, Chicago, IL, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1002\/rob.21568","article-title":"Analysis of human-robot interaction at the DARPA robotics challenge trials","volume":"32","author":"Yanco","year":"2015","journal-title":"J. Field Robot."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Murphy, R.R., and Schreckenghost, D. (2013, January 3\u20136). Survey of metrics for human-robot interaction. Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction, Tokyo, Japan.","DOI":"10.1109\/HRI.2013.6483569"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Murphy, R. (2014). Disaster Robotics, The MIT Press. Intelligent robotics and autonomous agents.","DOI":"10.7551\/mitpress\/9407.001.0001"},{"key":"ref_17","unstructured":"Zue, V. (2004). Eighty challenges facing speech input\/output technologies. Proc. Sound Sense, B179\u2013B195."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Devine, S., Rafferty, K., and Ferguson, S. (September, January 31). Real time robotic arm control using hand gestures with multiple end effectors. Proceedings of the 2016 UKACC 11th International Conference on Control (CONTROL), Belfast, UK.","DOI":"10.1109\/CONTROL.2016.7737564"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Du, G., Zhang, P., Mai, J., and Li, Z. (2012). Markerless Kinect-based hand tracking for robot teleoperation. Int. J. Adv. Robot. Syst., 9.","DOI":"10.5772\/50093"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Chen, S., and Ma, H. (2015). Hand Gesture Based Robot Control System Using Leap Motion. Int. Conf. Intell. Robot. Appl., 581\u2013591.","DOI":"10.1007\/978-3-319-22879-2_53"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sarkar, A., Patel, K.A., Ram, R.K.G., and Capoor, G.K. (2016, January 13\u201315). Gesture control of drone using a motion controller. Proceedings of the 2016 International Conference on Industrial Informatics and Computer Systems, CIICS 2016, Sharjah, United Arab Emirates.","DOI":"10.1109\/ICCSII.2016.7462401"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5898\/JHRI.5.2.Zhou","article-title":"A Comparative Study for Telerobotic Surgery Using Free Hand Gestures","volume":"5","author":"Zhou","year":"2016","journal-title":"J. Hum. Robot Interact."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1109\/TSMCC.2007.905819","article-title":"Human performance issues and user interface design for teleoperated robots","volume":"37","author":"Chen","year":"2007","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1177\/154193120304700123","article-title":"Manual versus speech input for unmanned aerial vehicle control station operations","volume":"Volume 47","author":"Draper","year":"2003","journal-title":"Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/MRA.2008.929929","article-title":"Stereoscopic human interfaces","volume":"15","author":"Ferre","year":"2008","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1080\/01691864.2014.1001787","article-title":"Development of the remote-controlled platform truck for handling heavy object and the remote control human interface for the disaster response","volume":"29","author":"Suwazono","year":"2015","journal-title":"Adv. Robot."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1080\/1463922021000054335","article-title":"The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task","volume":"5","author":"Kaber","year":"2004","journal-title":"Theor. Issues Ergon. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Muszynski, S., Stuckler, J., and Behnke, S. (2012, January 9\u201313). Adjustable autonomy for mobile teleoperation of personal service robots. Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication, Paris, France.","DOI":"10.1109\/ROMAN.2012.6343870"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1080\/1463922X.2013.815829","article-title":"Levels of automation: Effects of individual differences on wheelchair control performance and user acceptance","volume":"5","author":"Jipp","year":"2014","journal-title":"Theor. Issues Ergon. Sci."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Rosenberg, L.B. (1993). Virtual fixtures: Perceptual tools for telerobotic manipulation. Virtual Reality Annual International Symposium, IEEE.","DOI":"10.1109\/VRAIS.1993.380795"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Abbott, J., Marayong, P., and Okamura, A. (2007). Haptic virtual fixtures for robot-assisted manipulation. Robot. Res., 49\u201364.","DOI":"10.1007\/978-3-540-48113-3_5"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Park, Y.S., Zhao, X., and Korthals, S. (2014, January 5\u20137). Simulation of Augmented Telerobotic Operation. Proceedings of the 2014 International Symposium on Optomechatronic Technologies (ISOT), Seattle, WA, USA.","DOI":"10.1109\/ISOT.2014.65"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"DeJong, B.P., Faulring, E.L., Colgate, J.E., Peshkin, M.A., Kang, H., Park, Y.S., and Ewing, T.F. (2006). Lessons Learned From a Novel Teleoperation Testbed. Ind. Robot Int. J.","DOI":"10.1108\/01439910610659097"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1017\/S0263574709990415","article-title":"Virtual fixtures with autonomous error compensation for human-robot cooperative tasks","volume":"28","author":"Wahrburg","year":"2010","journal-title":"Robotica"},{"key":"ref_35","unstructured":"Ryd\u00e9n, F., Chizeck, H.J., Kosari, S.N., King, H., and Hannaford, B. (2014, January 12). Using kinect and a haptic interface for implementation of real-time virtual fixtures. Proceedings of the 2nd Workshop on RGB-D: Advanced Reasoning with Depth Cameras (in Conjunction with RSS 2011), Berkeley, CA, USA."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1507","DOI":"10.1080\/01691864.2014.962613","article-title":"Forbidden region virtual fixtures from streaming point clouds","volume":"28","author":"Kosari","year":"2014","journal-title":"Adv. Robot."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Kruusam\u00e4e, K., and Pryor, M. (2016, January 6\u20138). High-precision telerobot with human-centered variable perspective and scalable gestural interface. Proceedings of the 2016 9th International Conference on Human System Interactions, Portsmouth, UK.","DOI":"10.1109\/HSI.2016.7529630"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1331","DOI":"10.1109\/TSMCB.2009.2038357","article-title":"Comparative analysis of 3-D robot teleoperation interfaces with novice users","volume":"40","author":"Labonte","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part B Cybern."},{"key":"ref_39","unstructured":"Smart, W., Quigley, M., and Gerky, B. (2015). Programming Robots with ROS, O\u2019Reily Media."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6380","DOI":"10.3390\/s130506380","article-title":"Analysis of the accuracy and robustness of the leap motion controller","volume":"13","author":"Weichert","year":"2013","journal-title":"Sensors"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MRA.2011.2181749","article-title":"MoveIt!","volume":"19","author":"Chitta","year":"2012","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MRA.2012.2205651","article-title":"The Open Motion Planning Library","volume":"19","author":"Sucan","year":"2012","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Guimar\u00e3es, R.L., de Oliveira, A.S., Fabro, J.A., Becker, T., and Brenner, V.A. (2016). ROS Navigation: Concepts and Tutorial. Robot Operating System (ROS), Springer. Studies in Computational Intelligence.","DOI":"10.1007\/978-3-319-26054-9_6"},{"key":"ref_44","unstructured":"Huggins-Daines, D., Kumar, M., Chan, A., Black, A.W., Ravishankar, M., and Rudnicky, A.I. (2006, January 14\u201319). Pocketsphinx: A Free, Real-Time Continuous Speech Recognition System for Hand-Held Devices. Proceedings of the 2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings, Toulouse, France."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1773","DOI":"10.1017\/S0263574716000515","article-title":"An automatic switching approach to teleoperation of mobile-manipulator systems using virtual fixtures","volume":"35","author":"Wrock","year":"2017","journal-title":"Robotica"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1109\/TRO.2006.889486","article-title":"Improved techniques for grid mapping with rao-blackwellized particle filters","volume":"23","author":"Grisetti","year":"2007","journal-title":"IEEE Trans. Robot."},{"key":"ref_47","unstructured":"(2018, January 06). Sound_Play\u2014ROS Wiki. Available online: http:\/\/wiki.ros.org\/sound_play."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/MRA.2011.943230","article-title":"Interactive Markers: 3-D User Interfaces for ROS Applications [ROS Topics]","volume":"18","author":"Gossow","year":"2011","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Sharp, A., Kruusam\u00e4e, K., Ebersole, B., and Pryor, M. (2017, January 8\u201310). Semiautonomous dual-arm mobile manipulator system with intuitive supervisory user interfaces. Proceedings of the 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), Austin, TX, USA.","DOI":"10.1109\/ARSO.2017.8025195"},{"key":"ref_50","unstructured":"Pryor, M. (2017, January 5). Operator evaluation of a mobile robotic systems for inventory, inspection and contamination survey tasks in D&D environments. Proceedings of the Waste Management Symposia, Phoenix, AZ, USA."},{"key":"ref_51","unstructured":"United States Department of Energy (2017). Portsmouth Robotic Demo Final Report."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/7\/1\/9\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:53:29Z","timestamp":1760194409000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/7\/1\/9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,1]]},"references-count":51,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,3]]}},"alternative-id":["robotics7010009"],"URL":"https:\/\/doi.org\/10.3390\/robotics7010009","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,2,1]]}}}