{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T23:47:46Z","timestamp":1769730466756,"version":"3.49.0"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Sen. Netw."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Accurately reconstructing 3D hand poses is a pivotal element for numerous Human-Computer Interaction applications. In this work, we propose SonicHand, the first smartphone-based 3D hand pose reconstruction system using purely inaudible acoustic signals. SonicHand incorporates signal processing techniques and a deep learning framework to address a series of challenges. First, it encodes the topological information of the hand skeleton as prior knowledge and utilizes a deep learning model to realistically and smoothly reconstruct the hand poses. Second, the system employs adversarial training to enhance the generalization ability of our system to be deployed in a new environment or for a new user. Third, we adopt a hand tracking method based on channel impulse response estimation. It enables our system to handle the scenario where the hand performs gestures while moving arbitrarily as a whole. We conduct extensive experiments on a smartphone testbed to demonstrate the effectiveness and robustness of our system from various dimensions. The experiments involve 10 subjects performing up to 12 different hand gestures in three distinctive environments. When the phone is held in one of the user\u2019s hands, the proposed system can track joints with an average error of 18.64\u00a0mm.<\/jats:p>","DOI":"10.1145\/3677122","type":"journal-article","created":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T10:54:08Z","timestamp":1721127248000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic Signals"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0136-2663","authenticated-orcid":false,"given":"Shiyang","family":"Wang","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering, Purdue University, West Lafayette, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1352-7445","authenticated-orcid":false,"given":"Xingchen","family":"Wang","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Purdue University, West Lafayette, United States"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3194-7365","authenticated-orcid":false,"given":"Wenjun","family":"Jiang","sequence":"additional","affiliation":[{"name":"Samsung Research America, Mountain View, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9646-7099","authenticated-orcid":false,"given":"Chenglin","family":"Miao","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Iowa State University, Ames, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3329-3239","authenticated-orcid":false,"given":"Qiming","family":"Cao","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Purdue University, West Lafayette, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7485-6213","authenticated-orcid":false,"given":"Haoyu","family":"Wang","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Purdue University, West Lafayette, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9563-1398","authenticated-orcid":false,"given":"Ke","family":"Sun","sequence":"additional","affiliation":[{"name":"University of California, San Diego, La Jolla, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9691-9668","authenticated-orcid":false,"given":"Hongfei","family":"Xue","sequence":"additional","affiliation":[{"name":"The University of North Carolina at Charlotte, Charlotte, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7223-543X","authenticated-orcid":false,"given":"Lu","family":"Su","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Purdue University, West Lafayette, United States"}]}],"member":"320","published-online":{"date-parts":[[2024,8,26]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818072"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1958.tb03874.x"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01231-1_41"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430730"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2632090"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380091"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139544"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3550318"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00878"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.391"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.602"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01109"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3432215"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2016.2582924"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208331"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2015.7319426"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397306"},{"key":"e_1_3_3_19_2","first-page":"448","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning. 448\u2013456."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3241539.3241548"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372224.3380900"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3463519"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","unstructured":"Cem Keskin Furkan K\u0131ra\u00e7 Yunus Emre Kara and Lale Akarun. 2012. Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In Computer Vision\u2014ECCV 2012. Lecture Notes in Computer Science Vol. 7577. Springer 852\u2013863.","DOI":"10.1007\/978-3-642-33783-3_61"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298869"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380139"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1039\/C4NR03295K"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3495243.3560527"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3384419.3430780"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM42981.2021.9488693"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3478097"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2014.2306177"},{"issue":"7","key":"e_1_3_3_32_2","first-page":"2620","article-title":"UltraGesture: Fine-grained gesture sensing and recognition","volume":"21","author":"Ling Kang","year":"2020","unstructured":"Kang Ling, Haipeng Dai, Yuntang Liu, Alex X. Liu, Wei Wang, and Qing Gu. 2020. UltraGesture: Fine-grained gesture sensing and recognition. IEEE Transactions on Mobile Computing 21, 7 (2020), 2620\u20132636.","journal-title":"IEEE Transactions on Mobile Computing"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM42981.2021.9488768"},{"key":"e_1_3_3_34_2","article-title":"Adversary helps: Gradient-based device-free domain-independent gesture recognition","author":"Liu Jianwei","year":"2020","unstructured":"Jianwei Liu, Jinsong Han, Feng Lin, and Kui Ren. 2020. Adversary helps: Gradient-based device-free domain-independent gesture recognition. arXiv preprint arXiv:2004.03961 (2020).","journal-title":"arXiv preprint arXiv:2004.03961"},{"issue":"3","key":"e_1_3_3_35_2","first-page":"1","article-title":"Leveraging the properties of mmWave signals for 3D finger motion tracking for interactive IoT applications","volume":"6","author":"Liu Yilin","year":"2022","unstructured":"Yilin Liu, Shijia Zhang, Mahanth Gowda, and Srihari Nelakuditi. 2022. Leveraging the properties of mmWave signals for 3D finger motion tracking for interactive IoT applications. Proceedings of the ACM on Measurement and Analysis of Computing Systems 6, 3 (2022), 1\u201328.","journal-title":"Proceedings of the ACM on Measurement and Analysis of Computing Systems"},{"key":"e_1_3_3_36_2","article-title":"Towards deep learning models resistant to adversarial attacks","author":"Madry Aleksander","year":"2017","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).","journal-title":"arXiv preprint arXiv:1706.06083"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2973750.2973755"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3210240.3210325"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3300061.3345439"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2016.7759492"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00013"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858580"},{"key":"e_1_3_3_43_2","volume-title":"Proceedings of the 6th International Conference on Sensor Device Technologies and Applications","author":"O\u2019Flynn Brendan","year":"2015","unstructured":"Brendan O\u2019Flynn, J. Torres Sanchez, James Connolly, Joan Condell, Kevin Curran, Philip Gardiner, and Barry Downes. 2015. Integrated smart glove for hand motion monitoring. In Proceedings of the 6th International Conference on Sensor Device Technologies and Applications."},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.5244\/C.25.101"},{"key":"e_1_3_3_45_2","article-title":"PyTorch: An imperative style, high-performance deep learning library","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS\u201919). 8026\u20138037.","journal-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS\u201919)."},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/1322263.1322265"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892286"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/18.144727"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.145"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3084041.3084067"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534605"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971736"},{"key":"e_1_3_3_53_2","article-title":"A DIRT-T approach to unsupervised domain adaptation","author":"Shu Rui","year":"2018","unstructured":"Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. 2018. A DIRT-T approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.08735 (2018).","journal-title":"arXiv preprint arXiv:1802.08735"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.494"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417768"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58520-4_13"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3241539.3241568"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.490"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.380"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925965"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511807213"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00901"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3081333.3081340"},{"key":"e_1_3_3_65_2","article-title":"Generalizing to unseen domains via adversarial data augmentation","author":"Volpi Riccardo","year":"2018","unstructured":"Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C. Duchi, Vittorio Murino, and Silvio Savarese. 2018. Generalizing to unseen domains via adversarial data augmentation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS\u201918). 5339\u20135349.","journal-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS\u201918)."},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01111"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3550336"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300248"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2022.3230665"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00555"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3463522"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM48880.2022.9796906"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/2789168.2790093"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/2973750.2973764"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2020.3032278"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.3390\/s130506380"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/TAU.1967.1161901"},{"key":"e_1_3_3_78_2","unstructured":"Wikipedia. 2022. Quaternions and Spatial Rotation. Retrieved October 29 2022 from http:\/\/en.wikipedia.org\/w\/index.php?title=Quaternions%20and%20spatial%20rotation&oldid=1117495750"},{"key":"e_1_3_3_79_2","unstructured":"Wikipedia. 2023. Pixel 2. Retrieved January 17 2023 from http:\/\/en.wikipedia.org\/w\/index.php?title=Pixel%202&oldid=1132819321"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415897"},{"key":"e_1_3_3_81_2","article-title":"Empirical evaluation of rectified activations in convolutional network","author":"Xu Bing","year":"2015","unstructured":"Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015).","journal-title":"arXiv preprint arXiv:1505.00853"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.429"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581264"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458864.3467679"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01011"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/2742647.2742662"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3081333.3081356"},{"key":"e_1_3_3_88_2","article-title":"Unsupervised domain adaptation for device-free gesture recognition","author":"Zhang Bin-Bin","year":"2021","unstructured":"Bin-Bin Zhang, Dongheng Zhang, Yadong Li, Yang Hu, and Yan Chen. 2021. Unsupervised domain adaptation for device-free gesture recognition. arXiv preprint arXiv:2111.10602 (2021).","journal-title":"arXiv preprint arXiv:2111.10602"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3241539.3241570"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2021.3114309"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/2307636.2307638"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00768"},{"key":"e_1_3_3_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230543.3230579"},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3307334.3326081"},{"issue":"2","key":"e_1_3_3_95_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3534587","article-title":"Learning on the rings: Self-supervised 3D finger motion tracking using wearable sensors","volume":"6","author":"Zhou Hao","year":"2022","unstructured":"Hao Zhou, Taiting Lu, Yilin Liu, Shijia Zhang, and Mahanth Gowda. 2022. Learning on the rings: Self-supervised 3D finger motion tracking using wearable sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1\u201331.","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM48880.2022.9796782"},{"key":"e_1_3_3_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.525"}],"container-title":["ACM Transactions on Sensor Networks"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3677122","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3677122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:06:17Z","timestamp":1750291577000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3677122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,26]]},"references-count":96,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3677122"],"URL":"https:\/\/doi.org\/10.1145\/3677122","relation":{},"ISSN":["1550-4859","1550-4867"],"issn-type":[{"value":"1550-4859","type":"print"},{"value":"1550-4867","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,26]]},"assertion":[{"value":"2023-08-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}