{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:31:37Z","timestamp":1753893097261,"version":"3.41.2"},"reference-count":34,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,1,3]],"date-time":"2025-01-03T00:00:00Z","timestamp":1735862400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>We present WearMoCap, an open-source library to track the human pose from smartwatch sensor data and leveraging pose predictions for ubiquitous robot control. WearMoCap operates in three modes: 1) a Watch Only mode, which uses a smartwatch only, 2) a novel Upper Arm mode, which utilizes the smartphone strapped onto the upper arm and 3) a Pocket mode, which determines body orientation from a smartphone in any pocket. We evaluate all modes on large-scale datasets consisting of recordings from up to 8 human subjects using a range of consumer-grade devices. Further, we discuss real-robot applications of underlying works and evaluate WearMoCap in handover and teleoperation tasks, resulting in performances that are within 2 cm of the accuracy of the gold-standard motion capture system. Our Upper Arm mode provides the most accurate wrist position estimates with a Root Mean Squared prediction error of 6.79 cm. To evaluate WearMoCap in more scenarios and investigate strategies to mitigate sensor drift, we publish the WearMoCap system with thorough documentation as open source. The system is designed to foster future research in smartwatch-based motion capture for robotics applications where ubiquity matters. <jats:ext-link>www.github.com\/wearable-motion-capture<\/jats:ext-link>.<\/jats:p>","DOI":"10.3389\/frobt.2024.1478016","type":"journal-article","created":{"date-parts":[[2025,1,3]],"date-time":"2025-01-03T05:10:19Z","timestamp":1735881019000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["WearMoCap: multimodal pose tracking for ubiquitous robot control using a smartwatch"],"prefix":"10.3389","volume":"11","author":[{"given":"Fabian C.","family":"Weigend","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Neelesh","family":"Kumar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Oya","family":"Aran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Heni","family":"Ben Amor","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,1,3]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/MeMeA.2018.8438623","article-title":"Evaluation of wearable imu performance for orientation estimation and motion tracking","volume-title":"2018 IEEE international symposium on medical measurements and applications (MeMeA)","author":"Beange","year":"2018"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1706","DOI":"10.1109\/TRO.2023.3236952","article-title":"Teleoperation of humanoid robots: a survey","volume":"39","author":"Darvish","year":"2023","journal-title":"IEEE Trans. Robotics"},{"key":"B4","doi-asserted-by":"publisher","first-page":"103275","DOI":"10.1016\/j.cviu.2021.103275","article-title":"A review of 3d human pose estimation algorithms for markerless motion capture","volume":"212","author":"Desmarais","year":"2021","journal-title":"Comput. Vis. Image Underst."},{"key":"B5","doi-asserted-by":"crossref","DOI":"10.1145\/3586183.3606821","article-title":"Smartposer: arm pose estimation with a smartphone and smartwatch using uwb and imu data","volume-title":"Proceedings of the 36th annual ACM symposium on user interface software and technology","author":"DeVrio","year":"2023"},{"key":"B3","unstructured":"Humanplus: humanoid shadowing and imitation from humans\n          \n          \n            \n              Fu\n              Z.\n            \n            \n              Zhao\n              Q.\n            \n            \n              Wu\n              Q.\n            \n            \n              Wetzstein\n              G.\n            \n            \n              Finn\n              C.\n            \n          \n          \n          2024"},{"key":"B6","first-page":"1050","article-title":"Dropout as a bayesian approximation: representing model uncertainty in deep learning","author":"Gal","year":"2016"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-023-01095-w","article-title":"Analysis and perspectives on the ana avatar xprize competition","author":"Hauser","year":"2024","journal-title":"Int. J. Soc. Robotics"},{"key":"B8","doi-asserted-by":"publisher","first-page":"6628320","DOI":"10.1155\/2021\/6628320","article-title":"Inertial-based human motion capture: a technical summary of current processing methodologies for spatiotemporal and kinematic measures","volume":"2021","author":"Hindle","year":"2021","journal-title":"Appl. Bionics Biomechanics"},{"key":"B9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3272127.3275108","article-title":"Deep inertial poser: learning to reconstruct human pose from sparse inertial measurements in real time","volume":"37","author":"Huang","year":"2018","journal-title":"ACM Trans. Graph. (TOG)"},{"key":"B10","doi-asserted-by":"crossref","first-page":"1965","DOI":"10.1109\/IROS.2017.8206016","article-title":"Human motion estimation on lie groups using imu measurements","volume-title":"2017 IEEE\/RSJ international conference on intelligent robots and systems (IROS)","author":"Joukov","year":"2017"},{"key":"B11","first-page":"6126","article-title":"Smartwatch-based driver alertness monitoring with wearable motion and physiological sensor","volume-title":"2015 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC)","author":"Lee","year":"2015"},{"key":"B12","first-page":"1091","article-title":"Mocap everyone everywhere: lightweight motion capture with smartwatches and a head-mounted camera","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Lee","year":"2024"},{"key":"B13","doi-asserted-by":"publisher","first-page":"8953","DOI":"10.1109\/jiot.2021.3119328","article-title":"Real-time human motion capture based on wearable inertial sensor networks","volume":"9","author":"Li","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"B14","first-page":"120","article-title":"Real-time tracking of smartwatch orientation and location by multitask learning","author":"Liu","year":"2023"},{"key":"B15","doi-asserted-by":"crossref","first-page":"10212","DOI":"10.1109\/ICRA40945.2020.9196664","article-title":"Hand-worn haptic interface for drone teleoperation","volume-title":"2020 IEEE international conference on robotics and automation (ICRA)","author":"Macchini","year":"2020"},{"key":"B16","first-page":"449","article-title":"Real-time full-body motion capture from video and imus","author":"Malleson","year":"2017"},{"key":"B17","first-page":"1","article-title":"Imuposer: full-body pose estimation using imus in phones, watches, and earbuds","author":"Mollyn","year":"2023"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.17667\/riim.2018.1\/13","article-title":"Application of optitrack motion capture systems in human movement analysis: a systematic literature review","volume":"5","author":"Nagym\u00e1t\u00e9","year":"2018","journal-title":"Recent Innovations Mechatronics"},{"key":"B19","doi-asserted-by":"publisher","first-page":"5684","DOI":"10.1109\/access.2024.3350338ACCESS.2024.3350338","article-title":"A decade of progress in human motion recognition: a comprehensive survey from 2010 to 2020","volume":"12","author":"Noh","year":"2024","journal-title":"IEEE Access"},{"key":"B20","first-page":"670","article-title":"Design and implementation of imu-based human arm motion capture system","author":"Prayudi","year":"2012"},{"key":"B21","first-page":"155","article-title":"Design and development of a real-time, low-cost imu based human motion capture system","author":"Raghavendra","year":"2017"},{"key":"B22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3570731","article-title":"Robotic vision for human-robot interaction and collaboration: a survey and systematic review","volume":"12","author":"Robinson","year":"2023","journal-title":"J. Hum.-Robot Interact."},{"key":"B23","first-page":"1","article-title":"Xsens mvn: full 6dof human motion tracking using miniature inertial sensors","volume":"1","author":"Roetenberg","year":"2009","journal-title":"Xsens Motion Technol. BV, Tech. Rep."},{"key":"B24","doi-asserted-by":"publisher","first-page":"3082","DOI":"10.1109\/tbme.2023.3275775","article-title":"Markerless motion tracking with noisy video and imu data","volume":"70","author":"Shin","year":"2023","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B25","doi-asserted-by":"publisher","first-page":"109820","DOI":"10.1016\/j.jbiomech.2020.1098202020.109820","article-title":"A comparison of currently available optoelectronic motion capture systems","volume":"106","author":"Topley","year":"2020","journal-title":"J. Biomechanics"},{"key":"B26","doi-asserted-by":"publisher","first-page":"601","DOI":"10.1007\/s10514-019-09889-6","article-title":"Humans interacting with multi-robot systems: a natural affect-based approach","volume":"44","author":"Villani","year":"","journal-title":"Aut. Robots"},{"key":"B27","doi-asserted-by":"publisher","first-page":"13047","DOI":"10.1109\/JSEN.2020.3001635","article-title":"Wearable devices for the assessment of cognitive effort for human\u2013robot interaction","volume":"20","author":"Villani","year":"","journal-title":"IEEE Sensors J."},{"key":"B28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3597623","article-title":"Virtual, augmented, and mixed reality for human-robot interaction: a survey and virtual design element taxonomy","volume":"12","author":"Walker","year":"2023","journal-title":"J. Hum.-Robot Interact."},{"key":"B29","first-page":"4193","article-title":"AprilTag 2: efficient and robust fiducial detection","volume-title":"Proceedings of the IEEE\/RSJ international conference on intelligent robots and systems (IROS)","author":"Wang","year":"2016"},{"key":"B30","first-page":"7152","article-title":"Real-time limb motion tracking with a single imu sensor for physical therapy exercises","volume-title":"2021 43rd annual international conference of the IEEE engineering in medicine and biology society (EMBC)","author":"Wei","year":"2021"},{"key":"B31","unstructured":"Probabilistic differentiable filters enable ubiquitous robot control with smartwatches\n          \n          \n            \n              Weigend\n              F. C.\n            \n            \n              Liu\n              X.\n            \n            \n              Amor\n              H. B."},{"key":"B32","doi-asserted-by":"crossref","first-page":"17800","DOI":"10.1109\/ICRA57147.2024.10610805","article-title":"iRoCo: intuitive robot control from anywhere using a smartwatch","volume-title":"2024 IEEE international conference on robotics and automation (ICRA)","author":"Weigend","year":"2024"},{"key":"B33","first-page":"3811","article-title":"Anytime, anywhere: human arm pose from smartwatch data for ubiquitous robot control and teleoperation","author":"Weigend","year":""},{"key":"B34","first-page":"389","article-title":"Neural learning enhanced teleoperation control of baxter robot using imu based motion capture","author":"Yang","year":"2016"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1478016\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,3]],"date-time":"2025-01-03T05:10:32Z","timestamp":1735881032000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2024.1478016\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,3]]},"references-count":34,"alternative-id":["10.3389\/frobt.2024.1478016"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2024.1478016","relation":{},"ISSN":["2296-9144"],"issn-type":[{"type":"electronic","value":"2296-9144"}],"subject":[],"published":{"date-parts":[[2025,1,3]]},"article-number":"1478016"}}