{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T05:28:24Z","timestamp":1781155704349,"version":"3.54.1"},"reference-count":43,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,12,4]],"date-time":"2023-12-04T00:00:00Z","timestamp":1701648000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Virtual Real."],"abstract":"<jats:p>Today\u2019s research on human-robot teaming requires the ability to test artificial intelligence (AI) algorithms for perception and decision-making in complex real-world environments. Field experiments, also referred to as experiments \u201cin the wild,\u201d do not provide the level of detailed ground truth necessary for thorough performance comparisons and validation. Experiments on pre-recorded real-world data sets are also significantly limited in their usefulness because they do not allow researchers to test the effectiveness of active robot perception and control or decision strategies in the loop. Additionally, research on large human-robot teams requires tests and experiments that are too costly even for the industry and may result in considerable time losses when experiments go awry. The novel Real-Time Human Autonomous Systems Collaborations (RealTHASC) facility at Cornell University interfaces real and virtual robots and humans with photorealistic simulated environments by implementing new concepts for the seamless integration of wearable sensors, motion capture, physics-based simulations, robot hardware and virtual reality (VR). The result is an extended reality (XR) testbed by which real robots and humans in the laboratory are able to experience virtual worlds, inclusive of virtual agents, through real-time visual feedback and interaction. VR body tracking by DeepMotion is employed in conjunction with the OptiTrack motion capture system to transfer every human subject and robot in the real physical laboratory space into a synthetic virtual environment, thereby constructing corresponding human\/robot avatars that not only mimic the behaviors of the real agents but also experience the virtual world through virtual sensors and transmit the sensor data back to the real human\/robot agent, all in real time. New cross-domain synthetic environments are created in RealTHASC using Unreal Engine\u2122, bridging the simulation-to-reality gap and allowing for the inclusion of underwater\/ground\/aerial autonomous vehicles, each equipped with a multi-modal sensor suite. The experimental capabilities offered by RealTHASC are demonstrated through three case studies showcasing mixed real\/virtual human\/robot interactions in diverse domains, leveraging and complementing the benefits of experimentation in simulation and in the real world.<\/jats:p>","DOI":"10.3389\/frvir.2023.1210211","type":"journal-article","created":{"date-parts":[[2023,12,4]],"date-time":"2023-12-04T14:57:01Z","timestamp":1701701821000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["RealTHASC\u2014a cyber-physical XR testbed for AI-supported real-time human autonomous systems collaborations"],"prefix":"10.3389","volume":"4","author":[{"given":"Andre","family":"Paradise","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sushrut","family":"Surve","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jovan C.","family":"Menezes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Madhav","family":"Gupta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vaibhav","family":"Bisht","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyung Rak","family":"Jang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cong","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suming","family":"Qiu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junyi","family":"Dong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jane","family":"Shin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Silvia","family":"Ferrari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,12,4]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"724798","DOI":"10.3389\/frobt.2021.724798","article-title":"Augmented reality meets artificial intelligence in robotics: A systematic review","volume":"8","author":"Bassyouni","year":"2021","journal-title":"Front. Robotics AI"},{"key":"B2","article-title":"The opencv library","author":"Bradski","year":"2000","journal-title":"Dr. Dobb\u2019s J. Softw. Tools"},{"key":"B3","first-page":"1302","article-title":"Realtime multi-person 2d pose estimation using part affinity fields","author":"Cao","year":"2017"},{"key":"B4","first-page":"458","article-title":"Confidence level estimation in multi-target classification problems","author":"Chang","year":"2018"},{"key":"B5","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1109\/TRO.2015.2400731","article-title":"Occlusion-based cooperative transport with a swarm of miniature mobile robots","volume":"31","author":"Chen","year":"2015","journal-title":"IEEE Trans. Robotics"},{"key":"B6","doi-asserted-by":"publisher","first-page":"e1907856118","DOI":"10.1073\/pnas.1907856118","article-title":"On the use of simulation in robotics: opportunities, challenges, and suggestions for moving forward","volume":"118","author":"Choi","year":"2021","journal-title":"Proc. Natl. Acad. Sci."},{"key":"B7","volume-title":"DeepMotion SDK - virtual reality tracking","year":"2023"},{"key":"B8","first-page":"3161","article-title":"Robothor: an open simulation-to-real embodied ai platform","author":"Deitke","year":"2020"},{"key":"B9","first-page":"1","article-title":"Carla: an open urban driving simulator","author":"Dosovitskiy","year":"2017"},{"key":"B10","volume-title":"Unreal engine","year":"2019"},{"key":"B11","first-page":"4397","article-title":"Simulation tools for model-based robotics: comparison of bullet, havok, MuJoCo, ODE and PhysX","author":"Erez","year":"2015"},{"key":"B12","volume-title":"Information-driven planning and control","author":"Ferrari","year":"2021"},{"key":"B13","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1016\/S0921-8890(02)00372-X","article-title":"A survey of socially interactive robots","volume":"42","author":"Fong","year":"2003","journal-title":"Robotics Aut. Syst."},{"key":"B14","doi-asserted-by":"publisher","first-page":"10336","DOI":"10.3390\/su131810336","article-title":"Digital twin for fanuc robots: industrial robot programming and simulation using virtual reality","volume":"13","author":"Garg","year":"2021","journal-title":"Sustainability"},{"key":"B15","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1016\/j.ifacol.2019.01.062","article-title":"Video-guided camera control for target tracking and following","volume":"51","author":"Gemerek","year":"2019","journal-title":"IFAC-PapersOnLine"},{"key":"B16","volume-title":"Google cloud speech API","year":"2022"},{"key":"B17","first-page":"6941","article-title":"Flightgoggles: photorealistic sensor simulation for perception-driven robotics using photogrammetry and virtual reality","author":"Guerra","year":"2019"},{"key":"B18","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1016\/j.robot.2018.02.019","article-title":"An innovative tri-rotor drone and associated distributed aerial drone swarm control","volume":"103","author":"Hu","year":"2018","journal-title":"Robotics Aut. Syst."},{"key":"B19","doi-asserted-by":"publisher","first-page":"549360","DOI":"10.3389\/frobt.2021.549360","article-title":"Sigverse: A cloud-based vr platform for research on multimodal human-robot interaction","volume":"8","author":"Inamura","year":"2021","journal-title":"Front. Robotics AI"},{"key":"B20","first-page":"635","article-title":"Integration of gps and dead-reckoning navigation systems","volume-title":"Proc. Of the vehicle navigation and information systems conference, 1991","author":"Kao","year":"1991"},{"key":"B21","first-page":"2149","article-title":"Design and use paradigms for gazebo, an open-source multi-robot simulator","volume-title":"Proc. Of the 2004 IEEE\/RSJ international conference on intelligent robots and systems (IROS)(IEEE cat. No. 04CH37566)","author":"Koenig","year":"2004"},{"key":"B22","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1007\/s10846-014-0041-x","article-title":"A practical multirobot localization system","volume":"76","author":"Krajn\u00edk","year":"2014","journal-title":"J. Intelligent Robotic Syst."},{"key":"B23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3477391","article-title":"Coordinating human-robot teams with dynamic and stochastic task proficiencies","volume":"11","author":"Liu","year":"2021","journal-title":"ACM Trans. Human-Robot Interact. (THRI)"},{"key":"B24","doi-asserted-by":"publisher","first-page":"5","DOI":"10.5772\/5618","article-title":"Cyberbotics ltd. WebotsTM: professional mobile robot simulation","volume":"1","author":"Michel","year":"2004","journal-title":"Int. J. Adv. Robotic Syst."},{"key":"B25","doi-asserted-by":"crossref","first-page":"948","DOI":"10.1109\/SII.2017.8279345","article-title":"Cloud-based multimodal human-robot interaction simulator utilizing ROS and Unity frameworks","volume-title":"2017 IEEE\/SICE international symposium on system integration (SII)","author":"Mizuchi","year":"2017"},{"key":"B26","first-page":"470","article-title":"A simulator for human-robot interaction in virtual reality","author":"Murnane","year":"2021"},{"key":"B27","first-page":"255","article-title":"Analysis and design of human-robot swarm interaction in firefighting","volume-title":"Proc. Of RO-MAN 2008-the 17th IEEE international symposium on robot and human interactive communication","author":"Naghsh","year":"2008"},{"key":"B28","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1109\/MPRV.2005.13","article-title":"Human-robot teaming for search and rescue","volume":"4","author":"Nourbakhsh","year":"2005","journal-title":"IEEE Pervasive Comput."},{"key":"B29","doi-asserted-by":"publisher","first-page":"848065","DOI":"10.3389\/fnbot.2022.848065","article-title":"Editorial: active vision and perception in human-robot collaboration","volume":"16","author":"Ognibene","year":"2022","journal-title":"Front. Neurorobotics"},{"key":"B30","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1007\/978-3-319-50115-4_28","article-title":"Integrated intelligence for human-robot teams","volume-title":"Proc. Of the 2016 international symposium on experimental robotics","author":"Oh","year":"2017"},{"key":"B31","doi-asserted-by":"publisher","first-page":"6","DOI":"10.3390\/machines5010006","article-title":"Perception, planning, control, and coordination for autonomous vehicles","volume":"5","author":"Pendleton","year":"2017","journal-title":"Machines"},{"key":"B32","first-page":"8494","article-title":"Virtualhome: simulating household activities via programs","author":"Puig","year":"2018"},{"key":"B33","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1007\/978-3-319-49409-8_75","article-title":"Unrealcv: connecting computer vision to unreal engine","volume-title":"Proc. Of the computer vision \u2013 ECCV 2016 workshops","author":"Qiu","year":"2016"},{"key":"B34","doi-asserted-by":"publisher","first-page":"721933","DOI":"10.3389\/frvir.2021.721933","article-title":"The combination of artificial intelligence and extended reality: A systematic review","volume":"2","author":"Reiners","year":"2021","journal-title":"Front. Virtual Real."},{"key":"B35","first-page":"160","article-title":"High-frequency\/low-frequency synthetic aperture sonar","author":"Sammelmann","year":"1997"},{"key":"B36","first-page":"621","article-title":"Airsim: high-fidelity visual and physical simulation for autonomous vehicles","author":"Shah","year":"2018"},{"key":"B37","first-page":"7520","article-title":"Igibson 1.0: A simulation environment for interactive tasks in large realistic scenes","author":"Shen","year":"2021"},{"key":"B38","first-page":"1","article-title":"Synthetic sonar image simulation with various seabed conditions for automatic target recognition","volume-title":"Proc. Of the OCEANS 2022, hampton roads","author":"Shin","year":""},{"key":"B39","doi-asserted-by":"publisher","first-page":"780","DOI":"10.1109\/JOE.2021.3119150","article-title":"Informative multiview planning for underwater sensors","volume":"47","author":"Shin","year":"","journal-title":"IEEE J. Ocean. Eng."},{"key":"B40","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1016\/j.procir.2021.11.057","article-title":"An architecture for sim-to-real and real-to-sim experimentation in robotic systems","volume":"104","author":"\u0160kulj","year":"2021","journal-title":"Procedia CIRP"},{"key":"B41","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1002\/rob.21816","article-title":"Cooperative autonomous search, grasping, and delivering in a treasure hunt scenario by a team of unmanned aerial vehicles","volume":"36","author":"Spurn\u1ef3","year":"2019","journal-title":"J. Field Robotics"},{"key":"B42","first-page":"4724","article-title":"Convolutional pose machines","author":"Wei","year":"2016"},{"key":"B43","first-page":"2724","article-title":"Deep learning feature extraction for target recognition and classification in underwater sonar images","author":"Zhu","year":"2017"}],"container-title":["Frontiers in Virtual Reality"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frvir.2023.1210211\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,4]],"date-time":"2023-12-04T14:57:07Z","timestamp":1701701827000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frvir.2023.1210211\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,4]]},"references-count":43,"alternative-id":["10.3389\/frvir.2023.1210211"],"URL":"https:\/\/doi.org\/10.3389\/frvir.2023.1210211","relation":{},"ISSN":["2673-4192"],"issn-type":[{"value":"2673-4192","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,4]]},"article-number":"1210211"}}