{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T19:35:39Z","timestamp":1766086539080,"version":"3.41.2"},"reference-count":42,"publisher":"ASME International","issue":"1","content-domain":{"domain":["asmedigitalcollection.asme.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2009,3,1]]},"abstract":"<jats:p>A multimodal teleoperation interface is introduced, featuring an integrated virtual reality (VR) based simulation augmented by sensors and image processing capabilities onboard the remotely operated vehicle. The proposed virtual reality interface fuses an existing VR model with live video feed and prediction states, thereby creating a multimodal control interface. VR addresses the typical limitations of video based teleoperation caused by signal lag and limited field of view, allowing the operator to navigate in a continuous fashion. The vehicle incorporates an onboard computer and a stereo vision system to facilitate obstacle detection. A vehicle adaptation system with a priori risk maps and a real-state tracking system enable temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle\u2019s teleoperated state. The system provides real time update of the virtual environment based on anomalies encountered by the vehicle. The VR based multimodal teleoperation interface is expected to be more adaptable and intuitive when compared with other interfaces.<\/jats:p>","DOI":"10.1115\/1.3086030","type":"journal-article","created":{"date-parts":[[2009,3,6]],"date-time":"2009-03-06T23:30:53Z","timestamp":1236382253000},"update-policy":"https:\/\/doi.org\/10.1115\/crossmarkpolicy-asme","source":"Crossref","is-referenced-by-count":10,"title":["Sensor Augmented Virtual Reality Based Teleoperation Using Mixed Autonomy"],"prefix":"10.1115","volume":"9","author":[{"given":"Muthukkumar S.","family":"Kadavasal","sequence":"first","affiliation":[{"name":"Mem. ASME"},{"name":"Human Computer Interaction Program, Virtual Reality Applications Center, Iowa State University, Ames, IA 50010"}]},{"given":"James H.","family":"Oliver","sequence":"additional","affiliation":[{"name":"Mem. ASME"},{"name":"Department of Mechanical Engineering, Virtual Reality Applications Center, Iowa State University, Ames, IA 50010"}]}],"member":"33","published-online":{"date-parts":[[2009,3,3]]},"reference":[{"issue":"1","key":"2019100323575146100_c1","first-page":"37","article-title":"An Immersive Approach to Command and Control","volume":"7","author":"Knutzon","journal-title":"Journal of Battlefield Technology"},{"issue":"5","key":"2019100323575146100_c2","doi-asserted-by":"crossref","first-page":"520","DOI":"10.1162\/105474699566431","article-title":"Assisting the Teleoperation of an Unmanned Underwater Vehicle Using a Synthetic Subsea Scenario","volume":"8","author":"Lin","journal-title":"Presence: Teleoperators and Virtual Environments"},{"key":"2019100323575146100_c3","unstructured":"Knutzon, J. S.\n          , 2003, \u201cTracking and Control Design for a Virtual Reality Aided Teleoperation System,\u201d MS thesis, Iowa State University, Ames, IA."},{"issue":"4","key":"2019100323575146100_c4","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1162\/105474602760204264","article-title":"Human Interaction With Levels of Automation and Decision-Aid Fidelity in the Supervisory Control of Multiple Simulated Unmanned Air Vehicles","volume":"11","author":"Ruff","journal-title":"Presence: Teleoperators and Virtual Environments"},{"issue":"2","key":"2019100323575146100_c5","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1109\/21.3462","article-title":"The Effectiveness of Supervisory Control Strategies in Scheduling Flexible Manufacturing Systems","volume":"18","author":"Dunkler","journal-title":"IEEE Trans. Syst. Man Cybern.","ISSN":"https:\/\/id.crossref.org\/issn\/0018-9472","issn-type":"print"},{"key":"2019100323575146100_c6","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1023\/A:1011295826834","article-title":"Vehicle Teleoperation Interfaces","volume":"11","author":"Fong","journal-title":"Auton. Rob.","ISSN":"https:\/\/id.crossref.org\/issn\/0929-5593","issn-type":"print"},{"key":"2019100323575146100_c7","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1023\/A:1011299910904","article-title":"Teleoperation User Interfaces for Mining Robotics","volume":"11","author":"Hainsworth","journal-title":"Auton. Rob.","ISSN":"https:\/\/id.crossref.org\/issn\/0929-5593","issn-type":"print"},{"issue":"2","key":"2019100323575146100_c8","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1109\/70.134272","article-title":"Force-Reflection and Shared Compliant Control in Operating Telemanipulators With Time Delay","volume":"8","author":"Kim","journal-title":"IEEE Trans. Rob. Autom.","ISSN":"https:\/\/id.crossref.org\/issn\/1042-296X","issn-type":"print"},{"issue":"10","key":"2019100323575146100_c9","first-page":"42","article-title":"Reach Forward: What Happened to the Idea That Execution Authority Should be Delegated to the Lowest Possible Level","volume":"85","author":"Grant","journal-title":"Air Force Magazine, Air Force Association"},{"key":"2019100323575146100_c10","unstructured":"Glumm, M., Kilduff, P., and Masley, A., 1992, \u201cA Study on the Effects of Lens Focal Length on Remote Driver Performance,\u201d Army Research Laboratory, Technical Report No. ARL-TR-25."},{"key":"2019100323575146100_c11","unstructured":"Walter, B. E.\n          , 2003, \u201cVirtual Reality Aided Teleoperation,\u201d MS thesis, Iowa State University, Ames, IA."},{"key":"2019100323575146100_c12","first-page":"238","article-title":"Sensor Rich Teleoperation of an Excavating Machine","volume-title":"Proceedings of Conference on Field and Service Robotics","author":"Jarvis"},{"key":"2019100323575146100_c13","first-page":"2855","article-title":"Ecological Displays for Robot Interaction: A New Perspective","volume-title":"Proceedings of Conference on Intelligent Robots and Systems","author":"Ricks"},{"issue":"1","key":"2019100323575146100_c14","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/MCG.2005.23","article-title":"Time Follower\u2019s Vision: A Teleoperation Interface With Past Images","volume":"25","author":"Sugimoto","journal-title":"IEEE Comput. Graphics Appl.","ISSN":"https:\/\/id.crossref.org\/issn\/0272-1716","issn-type":"print"},{"issue":"4","key":"2019100323575146100_c15","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1109\/TSMCA.2005.850602","article-title":"Task-Driven Camera Operations for Robotic Exploration","volume":"35","author":"Hughes","journal-title":"IEEE Trans. Syst. Man Cybern., Part A. Syst. Humans","ISSN":"https:\/\/id.crossref.org\/issn\/1083-4427","issn-type":"print"},{"first-page":"21","article-title":"Telerobotic Control Using Augmented Reality","author":"Milgram","key":"2019100323575146100_c16"},{"first-page":"186","article-title":"Remote Vehicle Maneuvering Using Augmented Reality","author":"Brujic-Okretic","key":"2019100323575146100_c17"},{"issue":"4","key":"2019100323575146100_c18","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1162\/105474602760204273","article-title":"Augmented Reality as a Tool to Aid the Telerobotic Exploration and Characterization of Remote Environments","volume":"11","author":"Lawson","journal-title":"Presence: Teleoperators and Virtual Environments"},{"issue":"5","key":"2019100323575146100_c19","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1162\/105474602320935856","article-title":"Assistance for Telepresence by Stereovision Based Augmented Reality and Interactivity in 3D Space","volume":"11","author":"Fuchs","journal-title":"Presence: Teleoperators and Virtual Environments"},{"key":"2019100323575146100_c20","first-page":"219","article-title":"Synthetic Vision System for Improving Unmanned Aerial Vehicle Operator Situation Awareness","volume-title":"Proceedings of SPIE Enhanced and Synthetic Vision","author":"Calhoun"},{"key":"2019100323575146100_c21","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1023\/A:1011212313630","article-title":"Advanced Interfaces for Vehicle Teleoperation: Collaborative Control, Sensor Fusion Displays, and Remote Driving Tools","volume":"11","author":"Fong","journal-title":"Auton. Rob.","ISSN":"https:\/\/id.crossref.org\/issn\/0929-5593","issn-type":"print"},{"article-title":"Novel Interfaces for Remote Driving: Gesture, Haptic and PDA","volume-title":"Proceedings of SPIE Telemanipulator and Telepresence Technologies VII","author":"Fong","key":"2019100323575146100_c22"},{"author":"Milgram","key":"2019100323575146100_c23","article-title":"Real World Teleoperation via Virtual Environment Modeling"},{"key":"2019100323575146100_c24","first-page":"243","article-title":"Cooperative Robot Teleoperation Through Virtual Reality Interfaces","volume-title":"Proceedings of the Sixth International Conference on Information Visualization","author":"Monferrer"},{"key":"2019100323575146100_c25","unstructured":"Hine, B. P., Stoker, C., Sims, M., Rasmussen, D., Hontalas, P., Fong, T., Steele, J., Barch, D., Andersen, D., Miles, E., and Nygren, E., 1994, \u201cThe Application of Telepresence and Virtual Reality to Subsea Exploration,\u201d Second Workshop on Mobile Robots for Subsea Environments."},{"first-page":"13","article-title":"Computer and Information Sciences: Advanced Mixed Reality Technologies for Surveillance and Risk Prevention Applications","author":"Thalmann","key":"2019100323575146100_c26"},{"key":"2019100323575146100_c27","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1002\/hfm.20025","article-title":"Control Gain Adaptation in Virtual Reality Mediated Human-Telerobot Interaction","volume":"15","author":"Sheik-Nainar","journal-title":"Int. J. Hum. Factors Manufact.","ISSN":"https:\/\/id.crossref.org\/issn\/1090-8471","issn-type":"print"},{"key":"2019100323575146100_c28","first-page":"240","article-title":"3D Visualization Technologies for Teleguided Robots","volume-title":"Proceedings of the ACM Symposium on Virtual Reality Software and Technology","author":"Livatino"},{"issue":"3","key":"2019100323575146100_c29","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1109\/TVCG.2006.42","article-title":"CAVE and Fishtank Virtual-Reality Displays: A Qualitative and Quantitative Comparison","volume":"12","author":"Demiralp","journal-title":"IEEE Trans. Vis. Comput. Graph.","ISSN":"https:\/\/id.crossref.org\/issn\/1077-2626","issn-type":"print"},{"author":"Walter","key":"2019100323575146100_c30","article-title":"VR Aided Control of UAVs"},{"first-page":"888","article-title":"Site Model Acquisition and Extension from Aerial Images","author":"Collins","key":"2019100323575146100_c31"},{"key":"2019100323575146100_c32","first-page":"59","article-title":"Virtual Reality Interfaces for Visualization and Control of Remote Vehicles","volume-title":"Autonomous Robots","author":"Nguyen"},{"key":"2019100323575146100_c33","first-page":"297","article-title":"Sensor Enhanced Virtual Reality Teleoperation in Dynamic Environment","volume-title":"Proceedings of the IEEE Virtual Reality Conference","author":"Kadavasal"},{"author":"Meier","key":"2019100323575146100_c34","article-title":"Sensor Fusion Based User Interface for Vehicle Teleoperation"},{"key":"2019100323575146100_c35","unstructured":"Kroll, C. V., and Roland, R. D. , Jr., 1970, \u201cA Preview-Predictor Model of Driver Behavior in Emergency Situations,\u201d Cornell Aeronautical Laboratory, Inc., Bureau of Public Roads, Traffic Systems Research Division, Report No. CAL VJ-2251-V-6."},{"article-title":"Advancements in 3D Interactive Devices for Virtual Environments","author":"Wormell","key":"2019100323575146100_c36","doi-asserted-by":"crossref","DOI":"10.1145\/769953.769959"},{"key":"2019100323575146100_c37","first-page":"580","article-title":"Stereopsis Method for Visually Impaired to Identify Obstacles Based on Distance","volume-title":"Proceedings of the Third International Conference on Image and Graphics","author":"Balakrishnan"},{"key":"2019100323575146100_c38","unstructured":"Zitnick, C., and Kanade, T., 1999, \u201cA Cooperative Algorithm for Stereo Matching and Occlusion Detection,\u201d Robotics Institute, Carnegie Mellon University, Technical Report No. CMU-RI-TR-99-35."},{"first-page":"492","article-title":"A Maximum-Flow Formulation of the N-Camera Stereo Correspondence Problem","author":"Roy","key":"2019100323575146100_c39"},{"key":"2019100323575146100_c40","unstructured":"Birchfield, S., and Tomasi, C., 1996, \u201cDepth Discontinuities by Pixel-to-Pixel Stereo,\u201d Stanford University, Technical Report No. STAN-CSTR-96-1573, pp. 1073\u20131083."},{"key":"2019100323575146100_c41","unstructured":"http:\/\/www.vrjuggler.org\/"},{"key":"2019100323575146100_c42","unstructured":"http:\/\/www.openscenegraph.org\/projects\/osg"}],"container-title":["Journal of Computing and Information Science in Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/doi\/10.1115\/1.3086030\/5775069\/014502_1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/doi\/10.1115\/1.3086030\/5775069\/014502_1.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,10,4]],"date-time":"2019-10-04T03:58:23Z","timestamp":1570161503000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article\/doi\/10.1115\/1.3086030\/398342\/Sensor-Augmented-Virtual-Reality-Based"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,3,1]]},"references-count":42,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2009,3,1]]}},"URL":"https:\/\/doi.org\/10.1115\/1.3086030","relation":{},"ISSN":["1530-9827","1944-7078"],"issn-type":[{"type":"print","value":"1530-9827"},{"type":"electronic","value":"1944-7078"}],"subject":[],"published":{"date-parts":[[2009,3,1]]},"article-number":"014502"}}