{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,20]],"date-time":"2026-09-20T19:47:12Z","timestamp":1789933632868,"version":"4.0.1"},"reference-count":34,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2024,8,5]],"date-time":"2024-08-05T00:00:00Z","timestamp":1722816000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recently, research has been conducted on mixed reality (MR), which provides immersive visualization and interaction experiences, and on mapping human motions directly onto a robot in a mixed reality (MR) space to achieve a high level of immersion. However, even though the robot is mapped onto the MR space, their surrounding environment is often not mapped sufficiently; this makes it difficult to comfortably perform tasks that require precise manipulation of the objects that are difficult to see from the human perspective. Therefore, we propose a system that allows users to operate a robot in real space by mapping the task environment around the robot on the MR space and performing operations within the MR space.<\/jats:p>","DOI":"10.3390\/s24155073","type":"journal-article","created":{"date-parts":[[2024,8,5]],"date-time":"2024-08-05T15:45:22Z","timestamp":1722872722000},"page":"5073","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Immersive Robot Teleoperation Based on User Gestures in Mixed Reality Space"],"prefix":"10.3390","volume":"24","author":[{"given":"Hibiki","family":"Esaki","sequence":"first","affiliation":[{"name":"Department of Mechatronics Engineering, Graduate School of Science and Technology, Meijo University, 501-1 Shiogamaguchi, Nagoya 468-8502, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7288-8008","authenticated-orcid":false,"given":"Kosuke","family":"Sekiyama","sequence":"additional","affiliation":[{"name":"Department of Mechatronics Engineering, Graduate School of Science and Technology, Meijo University, 501-1 Shiogamaguchi, Nagoya 468-8502, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Zhang, T., McCarthy, Z., Jowl, O., Lee, D., Chen, X., Goldberg, K., and Abbeel, P. (2018, January 21\u201325). Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8461249"},{"key":"ref_2","first-page":"2530","article-title":"Collaborative Work in Augmented Reality: A Survey","volume":"72","author":"Sereno","year":"2020","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.ijhcs.2019.05.011","article-title":"Revisiting Collaboration through Mixed Reality: The Evolution of Groupware","volume":"131","author":"Ens","year":"2019","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"181","DOI":"10.3934\/ElectrEng.2019.2.181","article-title":"A Systematic Review of the Current State of Collaborative Mixed Reality Technologies: 2013\u20132018","volume":"3","author":"Belen","year":"2019","journal-title":"AIMS Electron. Electr. Eng."},{"key":"ref_5","first-page":"1321","article-title":"A Taxonomy of Mixed Reality Visual Displays","volume":"E77-D","author":"Milgram","year":"1994","journal-title":"IEICE Trans. Inf. Syst."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Demian, A., Ostanin, M., and Klimchik, A. (2021, January 26\u201329). Dynamic Object Grasping in Human-Robot Cooperation Based on Mixed-Reality. Proceedings of the 2021 International Conference \u201cNonlinearity, Information and Robotics\u201d (NIR), Innopolis, Russia.","DOI":"10.1109\/NIR52917.2021.9665812"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.ifacol.2018.11.517","article-title":"Interactive Robot Programming Using Mixed-Reality","volume":"51","author":"Ostanin","year":"2018","journal-title":"IFAC PapersOnLine"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"695","DOI":"10.1016\/j.ifacol.2019.11.307","article-title":"Interactive Robots Control Using Mixed Reality","volume":"52","author":"Ostanin","year":"2019","journal-title":"IFAC-PapersOnLine"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/MRA.2021.3138384","article-title":"Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together","volume":"24","author":"Delmerico","year":"2022","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/THMS.2019.2960676","article-title":"A New Mixed-Reality-Based Teleoperation System for Telepresence and Maneuverability Enhancement","volume":"50","author":"Sun","year":"2020","journal-title":"IEEE Trans.-Hum.-Mach. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Esaki, H., and Sekiyama, K. (2023, January 6\u20139). Human-Robot Interaction System based on MR Object Manipulation. Proceedings of the 2023 62nd Annual Conference of the Society of Instrument and Control Engineers (SICE), Tsu, Japan.","DOI":"10.23919\/SICE59929.2023.10354194"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Norman, D.A., and Draper, S.W. (1986). User Centered System Design: New Perspectives on Human-Computer Interaction, L. Erlbaum Associates Inc.","DOI":"10.1201\/b15703"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1007\/s12541-017-0113-5","article-title":"Sensorless Force Feedback Joystick Control for Teleoperation of Construction Equipment","volume":"18","author":"Dinh","year":"2017","journal-title":"Int. J. Precis. Eng. Manuf."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1007\/s12541-017-0038-z","article-title":"Force Reflecting Joystick Control for Applications to Bilateral Teleoperation in Construction Machinery","volume":"18","author":"Truong","year":"2017","journal-title":"Int. J. Precis. Eng. Manuf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/s40648-020-00163-4","article-title":"Free Viewpoint Image Generation System Using Fisheye Cameras and a Laser Rangefinder for Indoor Robot Teleoperation","volume":"7","author":"Komatsu","year":"2020","journal-title":"ROBOMECH J."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1080\/01691864.2020.1813623","article-title":"Towards the Development of an Intuitive Teleoperation System for Human Support Robot Using a VR Device","volume":"34","author":"Nakanishi","year":"2020","journal-title":"Adv. Robot."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Meeker, C., Rasmussen, T., and Ciocarlie, M. (2018, January 21\u201325). Intuitive Hand Teleoperation by Novice Operators Using a Continuous Teleoperation Subspace. Proceedings of the IEEE International Conference on Robotics and Automation, Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8460506"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1016\/j.procir.2016.11.183","article-title":"Immersive Augmented Reality Environment for the Teleoperation of Maintenance Robots","volume":"61","author":"Yew","year":"2017","journal-title":"Procedia CIRP"},{"key":"ref_19","unstructured":"Ellis, S.R., Adelstein, B.D., and Welch, R.B. (October, January 29). Kinesthetic Compensation for Misalignment of Teleoperator Controls through Cross-Modal Transfer of Movement Coordinates. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Baltimore, MD, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ribeiro, L.G., Suominen, O.J., Durmush, A., Peltonen, S., Morales, E.R., and Gotchev, A. (2021). Retro-Reflective-Marker-Aided Target Pose Estimation in a Safety-Critical Environment. Appl. Sci., 11.","DOI":"10.3390\/app11010003"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1016\/j.promfg.2020.10.087","article-title":"Mixed Reality Interface for Improving Mobile Manipulator Teleoperation in Contamination Critical Applications","volume":"51","author":"Bejczy","year":"2017","journal-title":"Procedia Manuf."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"78213","DOI":"10.1109\/ACCESS.2020.2990080","article-title":"Study of Multimodal Interfaces and the Improvements on Teleoperation","volume":"8","author":"Triantafyllidis","year":"2020","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"5113","DOI":"10.1109\/TVCG.2021.3101545","article-title":"A Conceptual Model and Taxonomy for Collaborative Augmented Reality","volume":"28","author":"Marques","year":"2021","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1016\/j.cag.2021.08.006","article-title":"A Critical Analysis on Remote Collaboration Mediated by Augmented Reality: Making a Case for Improved Characterization and Evaluation of the Collaborative Process","volume":"102","author":"Marques","year":"2022","journal-title":"Comput. Graph."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1007\/s10015-020-00587-6","article-title":"Dual-Arm Robot Teleoperation Support with the Virtual World","volume":"2","author":"Nakamura","year":"2020","journal-title":"Artif. Life Robot."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Whitney, D., Rosen, E., Ullman, D., Phillips, E., and Tellex, S. (2018, January 1\u20135). ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots. Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593513"},{"key":"ref_27","unstructured":"Whitney, D., Rosen, E., Phillips, E., Konidaris, G., and Tellex, S. (2017, January 11\u201314). Comparing Robot Grasping Teleoperation across Desktop and Virtual Reality with ROS Reality. Proceedings of the Robotics Research: The 18th International Symposium ISRR, Puerto Varas, Chile."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Delpreto, J., Lipton, J.I., Sanneman, L., Fay, A.J., Fourie, C., Choi, C., and Rus, D. (August, January 31). Helping Robots Learn: A Human-Robot Master-Apprentice Model Using Demonstrations via Virtual Reality Teleoperation. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9196754"},{"key":"ref_29","first-page":"259","article-title":"Lunar Micro Rover Design for Exploration through Virtual Reality Tele-Operation","volume":"Volume 105","author":"Britton","year":"2015","journal-title":"Tracts in Advanced Robotics"},{"key":"ref_30","first-page":"3645","article-title":"Simple Online and Realtime Tracking with a Deep Association Metric","volume":"99","author":"Wojke","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2022","DOI":"10.1109\/TPAMI.2012.257","article-title":"USAC: A Universal Framework for Random Sample Consensus","volume":"35","author":"Raguram","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_32","unstructured":"Rusinkiewicz, S., and Levoy, M. (June, January 28). Efficient Variants of the ICP Algorithm. Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling, Quebec City, QC, Canada."},{"key":"ref_33","first-page":"105","article-title":"The Levenberg-Marquardt Algorithm: Implementation and Theory","volume":"Volume 630","year":"1978","journal-title":"Numerical Analysis"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"112753","DOI":"10.1109\/ACCESS.2023.3322933","article-title":"Robot Symbolic Motion Planning and Task Execution Based on Mixed Reality Operation","volume":"11","author":"Nakamura","year":"2023","journal-title":"IEEE Access"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/15\/5073\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:30:26Z","timestamp":1760110226000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/15\/5073"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,5]]},"references-count":34,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["s24155073"],"URL":"https:\/\/doi.org\/10.3390\/s24155073","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,5]]}}}