{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T07:10:25Z","timestamp":1769929825205,"version":"3.49.0"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,6,23]],"date-time":"2021-06-23T00:00:00Z","timestamp":1624406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2021,6,23]]},"abstract":"<jats:p>IMU based inertial tracking plays an indispensable role in many mobility centric tasks, such as robotic control, indoor navigation and virtual reality gaming. Despite its mature application in rigid machine mobility (e.g., robot and aircraft), tracking human users via mobile devices remains a fundamental challenge due to the intractable gait\/posture patterns. Recent data-driven models have tackled sensor drifting, one key issue that plagues inertial tracking. However, these systems still assume the devices are held or attached to the user body with a relatively fixed posture. In practice, natural body activities may rotate\/translate the device which may be mistaken as whole body movement. Such motion artifacts remain as the dominating factor that fails existing inertial tracing systems in practical uncontrolled settings.<\/jats:p>\n          <jats:p>Inspired by the observation that human heads induces far less intensive movement relative to the body during walking, compared to other parts, we propose a novel multi-stage sensor fusion pipeline called DeepIT, which realizes inertial tracking by synthesizing the IMU measurements from a smartphone and an associated earbud. DeepIT introduces a data-driven reliability aware attention model, which assesses the reliability of each IMU and opportunistically synthesizes their data to mitigate the impacts of motion noise. Furthermore, DeepIT uses a reliability aware magnetometer compensation scheme to combat the angular drifting problem caused by unrestricted motion artifacts. We validate DeepIT on the first large-scale inertial navigation dataset involving both smartphone and earbud IMUs. The evaluation results show that DeepIT achieves multiple folds of accuracy improvement on the challenging uncontrolled natural walking scenarios, compared with state-of-the-art closed-form and data-driven models.<\/jats:p>","DOI":"10.1145\/3463517","type":"journal-article","created":{"date-parts":[[2021,6,24]],"date-time":"2021-06-24T16:29:19Z","timestamp":1624552159000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":33,"title":["Robust Inertial Motion Tracking through Deep Sensor Fusion across Smart Earbuds and Smartphone"],"prefix":"10.1145","volume":"5","author":[{"given":"Jian","family":"Gong","sequence":"first","affiliation":[{"name":"School of Computer Science and Engineering, Central South University, China and University of California San Diego"}]},{"given":"Xinyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of California San Diego, United States"}]},{"given":"Yuanjun","family":"Huang","sequence":"additional","affiliation":[{"name":"University of California San Diego, United States"}]},{"given":"Ju","family":"Ren","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Central South University, ChangSha, China"}]},{"given":"Yaoxue","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, BNRist, Tsinghua University, China"}]}],"member":"320","published-online":{"date-parts":[[2021,6,24]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493432.2493449"},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"MA Brodie Alan Walmsley and Wyatt Page. 2008. The static accuracy and calibration of inertial measurement units for 3D orientation. (2008).  MA Brodie Alan Walmsley and Wyatt Page. 2008. The static accuracy and calibration of inertial measurement units for 3D orientation. (2008).","DOI":"10.1080\/10255840802326736"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Changhao Chen Xiaoxuan Lu Andrew Markham and Niki Trigoni. 2018. IONet: Learning to Cure the Curse of Drift in Inertial Odometry.  Changhao Chen Xiaoxuan Lu Andrew Markham and Niki Trigoni. 2018. IONet: Learning to Cure the Curse of Drift in Inertial Odometry.","DOI":"10.1609\/aaai.v32i1.12102"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2019.2960780"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01079"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.2966773"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.3390\/s150100715"},{"key":"e_1_2_1_8_1","volume-title":"Gague Kim, and Hyuntae Jeong.","author":"Chung Seungeun","year":"2019","unstructured":"Seungeun Chung , Jiyoun Lim , Kyoung Ju Noh , Gague Kim, and Hyuntae Jeong. 2019 . Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning. Sensors 19, 7 (2019). https:\/\/www.mdpi.com\/1424-8220\/19\/7\/1716 Seungeun Chung, Jiyoun Lim, Kyoung Ju Noh, Gague Kim, and Hyuntae Jeong. 2019. Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning. Sensors 19, 7 (2019). https:\/\/www.mdpi.com\/1424-8220\/19\/7\/1716"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/MLSP.2018.8516710"},{"key":"e_1_2_1_10_1","volume-title":"Multimodal Deep Learning for Robust RGB-D Object Recognition. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS).","author":"Eitel Andreas","year":"2015","unstructured":"Andreas Eitel , Jost Springenberg , Luciano Spinello , Martin Riedmiller , and Wolfram Burgard . 2015 . Multimodal Deep Learning for Robust RGB-D Object Recognition. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). Andreas Eitel, Jost Springenberg, Luciano Spinello, Martin Riedmiller, and Wolfram Burgard. 2015. Multimodal Deep Learning for Robust RGB-D Object Recognition. In IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)."},{"key":"e_1_2_1_11_1","unstructured":"eSense. 2020. eSense website. https:\/\/www.esense.io\/  eSense. 2020. eSense website. https:\/\/www.esense.io\/"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3345615.3361131"},{"key":"e_1_2_1_13_1","unstructured":"Google. 2020. tango phone. https:\/\/support.google.com\/faqs\/faq\/6029402?hl=en  Google. 2020. tango phone. https:\/\/support.google.com\/faqs\/faq\/6029402?hl=en"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793604"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/SAS.2009.4801796"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/WISP.2009.5286542"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2014.2382568"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364914554813"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430776"},{"key":"e_1_2_1_20_1","volume-title":"An efficient orientation filter for inertial and inertial\/magnetic sensor arrays. Report x-io and University of Bristol (UK) 25","author":"Madgwick Sebastian","year":"2010","unstructured":"Sebastian Madgwick . 2010. An efficient orientation filter for inertial and inertial\/magnetic sensor arrays. Report x-io and University of Bristol (UK) 25 ( 2010 ). Sebastian Madgwick. 2010. An efficient orientation filter for inertial and inertial\/magnetic sensor arrays. Report x-io and University of Bristol (UK) 25 (2010)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.2008.923738"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPIN.2015.7346767"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Sebastian M\u00fcnzner Philip Schmidt Attila Reiss Michael Hanselmann Rainer Stiefelhagen and Robert D\u00fcrichen. 2017. CNN-based sensor fusion techniques for multimodal human activity recognition.  Sebastian M\u00fcnzner Philip Schmidt Attila Reiss Michael Hanselmann Rainer Stiefelhagen and Robert D\u00fcrichen. 2017. CNN-based sensor fusion techniques for multimodal human activity recognition.","DOI":"10.1145\/3123021.3123046"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Armando Neto Douglas Macharet V\u00edctor Campos and Mario Campos. 2009. Adaptive complementary filtering algorithm for mobile robot localization. J. Braz. Comp. Soc. (2009).  Armando Neto Douglas Macharet V\u00edctor Campos and Mario Campos. 2009. Adaptive complementary filtering algorithm for mobile robot localization. J. Braz. Comp. Soc. (2009).","DOI":"10.1007\/BF03194503"},{"key":"e_1_2_1_25_1","volume-title":"Gyroscope drift correction algorithm for inertial measurement unit used in hand motion tracking. 2016 IEEE SENSORS","author":"Larnnithipong Nonnarit","year":"2016","unstructured":"Nonnarit O.- Larnnithipong and Armando Barreto . 2016. Gyroscope drift correction algorithm for inertial measurement unit used in hand motion tracking. 2016 IEEE SENSORS ( 2016 ), 1--3. Nonnarit O.-Larnnithipong and Armando Barreto. 2016. Gyroscope drift correction algorithm for inertial measurement unit used in hand motion tracking. 2016 IEEE SENSORS (2016), 1--3."},{"key":"e_1_2_1_26_1","volume-title":"2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). 1531--1536","author":"Patel N.","unstructured":"N. Patel , A. Choromanska , P. Krishnamurthy , and F. Khorrami . 2017. Sensor modality fusion with CNNs for UGV autonomous driving in indoor environments . In 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). 1531--1536 . N. Patel, A. Choromanska, P. Krishnamurthy, and F. Khorrami. 2017. Sensor modality fusion with CNNs for UGV autonomous driving in indoor environments. In 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). 1531--1536."},{"key":"e_1_2_1_27_1","volume-title":"STEAR: Robust Step Counting from Earables. https:\/\/doi.org\/10.1145\/3345615.3361133","author":"Prakash Jay","year":"2019","unstructured":"Jay Prakash , Zhijian Yang , Yu-Lin Wei , and Romit Choudhury . 2019 . STEAR: Robust Step Counting from Earables. https:\/\/doi.org\/10.1145\/3345615.3361133 10.1145\/3345615.3361133 Jay Prakash, Zhijian Yang, Yu-Lin Wei, and Romit Choudhury. 2019. STEAR: Robust Step Counting from Earables. https:\/\/doi.org\/10.1145\/3345615.3361133"},{"key":"e_1_2_1_28_1","unstructured":"Pytorch. 2020. Pytorch website. https:\/\/pytorch.org\/  Pytorch. 2020. Pytorch website. https:\/\/pytorch.org\/"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432221"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of IEEE CVPR.","author":"Qian Kun","year":"2020","unstructured":"Kun Qian , Shilin Zhu , Xinyu Zhang , and Li Erran Li . 2020 . Robust Multimodal Vehicle Detection in Adverse Weather using Complementary Lidar and Radar Signals . In Proceedings of IEEE CVPR. Kun Qian, Shilin Zhu, Xinyu Zhang, and Li Erran Li. 2020. Robust Multimodal Vehicle Detection in Adverse Weather using Complementary Lidar and Radar Signals. In Proceedings of IEEE CVPR."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2969170"},{"key":"e_1_2_1_32_1","article-title":"Strapdown Inertial Navigation Integration Algorithm Design Part 1","volume":"21","author":"Savage Paul G.","year":"1998","unstructured":"Paul G. Savage . 1998 . Strapdown Inertial Navigation Integration Algorithm Design Part 1 : Attitude Algorithms. Journal of Guidance, Control, and Dynamics 21 , 1 (1998). Paul G. Savage. 1998. Strapdown Inertial Navigation Integration Algorithm Design Part 1: Attitude Algorithms. Journal of Guidance, Control, and Dynamics 21, 1 (1998).","journal-title":"Journal of Guidance, Control, and Dynamics"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3241539.3241582"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2789168.2790099"},{"key":"e_1_2_1_35_1","unstructured":"J\u00fcrgen Sturm St\u00e9phane Magnenat Nikolas Engelhard Fran\u00e7ois Pomerleau Francis Colas Daniel Cremers Roland Siegwart and Wolfram Burgard. 2011. Towards a benchmark for RGB-D SLAM evaluation.  J\u00fcrgen Sturm St\u00e9phane Magnenat Nikolas Engelhard Fran\u00e7ois Pomerleau Francis Colas Daniel Cremers Roland Siegwart and Wolfram Burgard. 2011. Towards a benchmark for RGB-D SLAM evaluation."},{"key":"e_1_2_1_36_1","volume-title":"2015 IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP). IEEE, 1--6.","author":"Tian Qinglin","year":"2015","unstructured":"Qinglin Tian , Zoran Salcic , I Kevin , Kai Wang , and Yun Pan . 2015 . An enhanced pedestrian dead reckoning approach for pedestrian tracking using smartphones . In 2015 IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP). IEEE, 1--6. Qinglin Tian, Zoran Salcic, I Kevin, Kai Wang, and Yun Pan. 2015. An enhanced pedestrian dead reckoning approach for pedestrian tracking using smartphones. In 2015 IEEE Tenth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP). IEEE, 1--6."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858084"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the International Conference on Neural Information Processing Systems.","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , undefinedukasz Kaiser, and Illia Polosukhin . 2017 . Attention is All You Need . In Proceedings of the International Conference on Neural Information Processing Systems. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Proceedings of the International Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_39_1","volume-title":"LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation. In International Conference on Indoor Positioning and Indoor Navigation (IPIN).","author":"Wagstaff B.","unstructured":"B. Wagstaff and J. Kelly . 2018 . LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation. In International Conference on Indoor Positioning and Indoor Navigation (IPIN). B. Wagstaff and J. Kelly. 2018. LSTM-Based Zero-Velocity Detection for Robust Inertial Navigation. In International Conference on Indoor Positioning and Indoor Navigation (IPIN)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858223"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3380980"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3323679.3326513"},{"key":"e_1_2_1_43_1","volume-title":"RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, and New Methods. arXiv preprint arXiv:1905.12853","author":"Yan Hang","year":"2019","unstructured":"Hang Yan , Sachini Herath , and Yasutaka Furukawa . 2019. RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, and New Methods. arXiv preprint arXiv:1905.12853 ( 2019 ). Hang Yan, Sachini Herath, and Yasutaka Furukawa. 2019. RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, and New Methods. arXiv preprint arXiv:1905.12853 (2019)."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01261-8_38"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"S. Yao Y. Zhao H. Shao D. Liu S. Liu Y. Hao A. Piao S. Hu S. Lu and T. F. Abdelzaher. 2019. SADeepSense: Self-Attention Deep Learning Framework for Heterogeneous On-Device Sensors in Internet of Things Applications. In IEEE INFOCOM 2019 - IEEE Conference on Computer Communications. 1243--1251.  S. Yao Y. Zhao H. Shao D. Liu S. Liu Y. Hao A. Piao S. Hu S. Lu and T. F. Abdelzaher. 2019. SADeepSense: Self-Attention Deep Learning Framework for Heterogeneous On-Device Sensors in Internet of Things Applications. In IEEE INFOCOM 2019 - IEEE Conference on Computer Communications. 1243--1251.","DOI":"10.1109\/INFOCOM.2019.8737500"},{"key":"e_1_2_1_46_1","volume-title":"Electronics and Mobile Communication Conference (IEMCON). IEEE, 1--6.","author":"Yean Seanglidet","year":"2016","unstructured":"Seanglidet Yean , Bu Sung Lee , Chai Kiat Yeo , and Chan Hua Vun . 2016 . Algorithm for 3D orientation estimation based on Kalman filter and gradient descent. In 2016 IEEE 7th Annual Information Technology , Electronics and Mobile Communication Conference (IEMCON). IEEE, 1--6. Seanglidet Yean, Bu Sung Lee, Chai Kiat Yeo, and Chan Hua Vun. 2016. Algorithm for 3D orientation estimation based on Kalman filter and gradient descent. In 2016 IEEE 7th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON). IEEE, 1--6."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267242.3267286"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3463517","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3463517","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:28Z","timestamp":1750195888000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3463517"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,23]]},"references-count":47,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,6,23]]}},"alternative-id":["10.1145\/3463517"],"URL":"https:\/\/doi.org\/10.1145\/3463517","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,23]]},"assertion":[{"value":"2021-06-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}