{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T08:00:57Z","timestamp":1773734457501,"version":"3.50.1"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T00:00:00Z","timestamp":1616112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"d ERANET-COFUND (H2020) CHIST-ERA III, RadioSense."},{"name":"European Union\u00f0s Horizon 2020 research and innovation programme under the Marie Sk\u00f0odowska-Curie grant","award":["813999"],"award-info":[{"award-number":["813999"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2021,3,19]]},"abstract":"<jats:p>We introduce Pantomime, a novel mid-air gesture recognition system exploiting spatio-temporal properties of millimeter-wave radio frequency (RF) signals. Pantomime is positioned in a unique region of the RF landscape: mid-resolution mid-range high-frequency sensing, which makes it ideal for motion gesture interaction. We configure a commercial frequency-modulated continuous-wave radar device to promote spatial information over the temporal resolution by means of sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging set of 21 gestures articulated by 41 participants in two indoor environments, outperforming four state-of-the-art 3D point cloud recognizers. We further analyze the effect of the environment in 5 different indoor environments, the effect of articulation speed, angle, and the distance of the person up to 5m. We have publicly made available the collected mmWave gesture dataset consisting of nearly 22,000 gesture instances along with our radar sensor configuration, trained models, and source code for reproducibility. We conclude that pantomime is resilient to various input conditions and that it may enable novel applications in industrial, vehicular, and smart home scenarios.<\/jats:p>","DOI":"10.1145\/3448110","type":"journal-article","created":{"date-parts":[[2021,3,30]],"date-time":"2021-03-30T18:56:41Z","timestamp":1617130601000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":141,"title":["Pantomime"],"prefix":"10.1145","volume":"5","author":[{"given":"Sameera","family":"Palipana","sequence":"first","affiliation":[{"name":"Aalto University, Finland"}]},{"given":"Dariush","family":"Salami","sequence":"additional","affiliation":[{"name":"Aalto University, Finland"}]},{"given":"Luis A.","family":"Leiva","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}]},{"given":"Stephan","family":"Sigg","sequence":"additional","affiliation":[{"name":"Aalto University, Finland"}]}],"member":"320","published-online":{"date-parts":[[2021,3,30]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"ABB. 2020. YuMi - Creating an automated future together. (2020). https:\/\/new.abb.com\/products\/robotics\/industrial-robots\/irb-14000-yumi (accessed: 09.11.2020).  ABB. 2020. YuMi - Creating an automated future together. (2020). https:\/\/new.abb.com\/products\/robotics\/industrial-robots\/irb-14000-yumi (accessed: 09.11.2020)."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2015.7218525"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818072"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172962"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2942305"},{"key":"e_1_2_2_6_1","volume-title":"Proc. NIME. 283--286","author":"Bernardo Francisco","year":"2017","unstructured":"Francisco Bernardo , Nicholas Arner , and Paul Batchelor . 2017 . O soli mio: exploring millimeter wave radar for musical interaction . In Proc. NIME. 283--286 . Francisco Bernardo, Nicholas Arner, and Paul Batchelor. 2017. O soli mio: exploring millimeter wave radar for musical interaction. In Proc. NIME. 283--286."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2235592100"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157689.2157807"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2005.03.002"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/83.623193"},{"key":"e_1_2_2_11_1","unstructured":"Barrett Ens Aaron Quigley Hui-Shyong Yeo Pourang Irani Thammathip Piumsomboon and Mark Billinghurst. 2017. Exploring Mixed-Scale Gesture Interaction. In SIGGRAPH Asia Posters. 1--2.  Barrett Ens Aaron Quigley Hui-Shyong Yeo Pourang Irani Thammathip Piumsomboon and Mark Billinghurst. 2017. Exploring Mixed-Scale Gesture Interaction. In SIGGRAPH Asia Posters. 1--2."},{"key":"e_1_2_2_12_1","volume-title":"Proc. KDD. 226--231","author":"Ester Martin","year":"1996","unstructured":"Martin Ester , Hans-Peter Kriegel , J\u00f6rg Sander , and Xiaowei Xu . 1996 . A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise . In Proc. KDD. 226--231 . Martin Ester, Hans-Peter Kriegel, J\u00f6rg Sander, and Xiaowei Xu. 1996. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proc. KDD. 226--231."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00878"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.391"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.602"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.3390\/s140203702"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2807551"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1622176.1622203"},{"key":"e_1_2_2_19_1","volume-title":"Proc. BMVC. 1--13","author":"Kingkan Cherdsak","year":"2018","unstructured":"Cherdsak Kingkan , Joshua Owoyemi , and Koichi Hashimoto . 2018 . Point Attention Network for Gesture Recognition Using Point Cloud Data . In Proc. BMVC. 1--13 . Cherdsak Kingkan, Joshua Owoyemi, and Koichi Hashimoto. 2018. Point Attention Network for Gesture Recognition Using Point Cloud Data. In Proc. BMVC. 1--13."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2019.1572352"},{"key":"e_1_2_2_21_1","unstructured":"KUKA. 2020. Cobots in the industry. (2020). https:\/\/www.kuka.com\/en-de\/future-production\/industrie-4-0\/industrie-4-0-cobots-in-industry (accessed: 09.11.2020).  KUKA. 2020. Cobots in the industry. (2020). https:\/\/www.kuka.com\/en-de\/future-production\/industrie-4-0\/industrie-4-0-cobots-in-industry (accessed: 09.11.2020)."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.115"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406324.3410720"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971738"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00096"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925953"},{"key":"e_1_2_2_27_1","article-title":"A survey of applications and human motion recognition with Microsoft","volume":"29","author":"Lun Roanna","year":"2015","unstructured":"Roanna Lun and Wenbing Zhao . 2015 . A survey of applications and human motion recognition with Microsoft Kinect. Int. J. Pattern Recognit. Artif. Intell. 29 , 5 (2015), 1555008:1-48. Roanna Lun and Wenbing Zhao. 2015. A survey of applications and human motion recognition with Microsoft Kinect. Int. J. Pattern Recognit. Artif. Intell. 29, 5 (2015), 1555008:1-48.","journal-title":"Kinect. Int. J. Pattern Recognit. Artif. Intell."},{"key":"e_1_2_2_28_1","first-page":"1","article-title":"SignFi: Sign language recognition using wifi","volume":"2","author":"Ma Yongsen","year":"2018","unstructured":"Yongsen Ma , Gang Zhou , Shuangquan Wang , Hongyang Zhao , and Woosub Jung . 2018 . SignFi: Sign language recognition using wifi . ACM IMWUT 2 , 1 (2018), 1 -- 21 . Yongsen Ma, Gang Zhou, Shuangquan Wang, Hongyang Zhao, and Woosub Jung. 2018. SignFi: Sign language recognition using wifi. ACM IMWUT 2, 1 (2018), 1--21.","journal-title":"ACM IMWUT"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2014.7025313"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053152"},{"key":"e_1_2_2_31_1","volume-title":"Proc. AAAI. 1--8.","author":"Meng Zhen","year":"2020","unstructured":"Zhen Meng , Song Fu , Jie Yan , Hongyuan Liang , Anfu Zhou , Shilin Zhu , Huadong Ma , Jianhua Liu , and Ning Yang . 2020 . Gait Recognition for Co-existing Multiple People Using Millimeter Wave Sensing . In Proc. AAAI. 1--8. Zhen Meng, Song Fu, Jie Yan, Hongyuan Liang, Anfu Zhou, Shilin Zhu, Huadong Ma, Jianhua Liu, and Ning Yang. 2020. Gait Recognition for Co-existing Multiple People Using Millimeter Wave Sensing. In Proc. AAAI. 1--8."},{"key":"e_1_2_2_32_1","volume-title":"Proc. BMVC. 1--13","author":"Min Yuecong","year":"2019","unstructured":"Yuecong Min , Xiujuan Chai , Lei Zhao , and Xilin Chen . 2019 . FlickerNet: Adaptive 3D Gesture Recognition from Sparse Point Clouds . In Proc. BMVC. 1--13 . Yuecong Min, Xiujuan Chai, Lei Zhao, and Xilin Chen. 2019. FlickerNet: Adaptive 3D Gesture Recognition from Sparse Point Clouds. In Proc. BMVC. 1--13."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.456"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/LSENS.2018.2865165"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8460910"},{"key":"e_1_2_2_36_1","volume-title":"Pantomime dataset and source. (April","author":"Palipana Sameera","year":"2020","unstructured":"Sameera Palipana , Dariush Salami , Luis Leiva , and Stephan Sigg . 2020. Pantomime dataset and source. (April 2020 ). DOI:http:\/\/dx.doi.org\/10.5281\/zenodo.4459969 Sameera Palipana, Dariush Salami, Luis Leiva, and Stephan Sigg. 2020. Pantomime dataset and source. (April 2020). DOI:http:\/\/dx.doi.org\/10.5281\/zenodo.4459969"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1120212.1120391"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2015.08.004"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2500423.2500436"},{"key":"e_1_2_2_40_1","volume-title":"Proc. CVPR. 652--660","author":"Qi Charles R","year":"2017","unstructured":"Charles R Qi , Hao Su , Kaichun Mo , and Leonidas J Guibas . 2017 a. PointNet: Deep learning on point sets for 3D classification and segmentation . In Proc. CVPR. 652--660 . Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017a. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. CVPR. 652--660."},{"key":"e_1_2_2_41_1","volume-title":"Proc. NeurIPS. 5099--5108","author":"Qi Charles Ruizhongtai","year":"2017","unstructured":"Charles Ruizhongtai Qi , Li Yi , Hao Su , and Leonidas J Guibas . 2017 b. PointNet++: Deep hierarchical feature learning on point sets in a metric space . In Proc. NeurIPS. 5099--5108 . Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proc. NeurIPS. 5099--5108."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2750858.2804280"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-012-9356-9"},{"key":"e_1_2_2_44_1","volume-title":"Holm","author":"Richards Mark A.","year":"2010","unstructured":"Mark A. Richards , James A. Scheer , and William A . Holm . 2010 . Principles of Modern Radar: Basic Principles . Scitech Publishing . Mark A. Richards, James A. Scheer, and William A. Holm. 2010. Principles of Modern Radar: Basic Principles. Scitech Publishing."},{"key":"e_1_2_2_45_1","first-page":"300","article-title":"MIMO radar theory and experimental results","volume":"1","author":"Robey Frank C","year":"2004","unstructured":"Frank C Robey , Scott Coutts , Dennis Weikle , Jeffrey C McHarg , and Kevin Cuomo . 2004 . MIMO radar theory and experimental results . In Proc. ACSSC , Vol. 1. 300 -- 304 . Frank C Robey, Scott Coutts, Dennis Weikle, Jeffrey C McHarg, and Kevin Cuomo. 2004. MIMO radar theory and experimental results. In Proc. ACSSC, Vol. 1. 300--304.","journal-title":"Proc. ACSSC"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979341"},{"key":"e_1_2_2_47_1","doi-asserted-by":"crossref","unstructured":"D. Salami S. Palipana M. Kodali and S. Sigg. 2020. Motion Pattern Recognition in 4D Point Clouds. In Int. Works. on Machine Learning for Signal Processing (MLSP). 1--6.  D. Salami S. Palipana M. Kodali and S. Sigg. 2020. Motion Pattern Recognition in 4D Point Clouds. In Int. Works. on Machine Learning for Signal Processing (MLSP). 1--6.","DOI":"10.1109\/MLSP49062.2020.9231569"},{"key":"e_1_2_2_48_1","volume-title":"Proc. ISMAR-Adjunct. 427--427","author":"Sandor C.","unstructured":"C. Sandor and H. Nakamura . 2018. SoliScratch: A Radar Interface for Scratch DJs . In Proc. ISMAR-Adjunct. 427--427 . C. Sandor and H. Nakamura. 2018. SoliScratch: A Radar Interface for Scratch DJs. In Proc. ISMAR-Adjunct. 427--427."},{"key":"e_1_2_2_49_1","first-page":"1","article-title":"Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals","volume":"4","author":"Santhalingam Panneer Selvam","year":"2020","unstructured":"Panneer Selvam Santhalingam , Al Amin Hosain , Ding Zhang , Parth Pathak , Huzefa Rangwala , and Raja Kushalnagar . 2020 . Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals . ACM IMWUT 4 , 1 (2020), 1 -- 30 . Panneer Selvam Santhalingam, Al Amin Hosain, Ding Zhang, Parth Pathak, Huzefa Rangwala, and Raja Kushalnagar. 2020. Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals. ACM IMWUT 4, 1 (2020), 1--30.","journal-title":"ACM IMWUT"},{"key":"e_1_2_2_50_1","first-page":"10","article-title":"Non-Invasive Monitoring of Glucose Level Changes Utilizing a mm-Wave Radar System","volume":"19","author":"Shaker G.","year":"2018","unstructured":"G. Shaker , K. Smith , A. E. Omer , S. Liu , C. Csech , U. Wadhwa , S. Safavi-Naeini , and R. Hughson . 2018 . Non-Invasive Monitoring of Glucose Level Changes Utilizing a mm-Wave Radar System . IJMHCI 19 , 3 (2018), 10 -- 29 . G. Shaker, K. Smith, A. E. Omer, S. Liu, C. Csech, U. Wadhwa, S. Safavi-Naeini, and R. Hughson. 2018. Non-Invasive Monitoring of Glucose Level Changes Utilizing a mm-Wave Radar System. IJMHCI 19, 3 (2018), 10--29.","journal-title":"IJMHCI"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3349624.3356768"},{"key":"e_1_2_2_52_1","volume-title":"Portigal","author":"Stine John A.","year":"2004","unstructured":"John A. Stine and David L . Portigal . 2004 . Spectrum 101: An Introduction to Spectrum Management. Technical Report. MITRE Corporation . John A. Stine and David L. Portigal. 2004. Spectrum 101: An Introduction to Spectrum Management. Technical Report. MITRE Corporation."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.114"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1461551.1461591"},{"key":"e_1_2_2_55_1","volume-title":"Proc. IFIP Congress. 506--508","author":"Sutherland I. E.","year":"1965","unstructured":"I. E. Sutherland . 1965 . The ultimate display . In Proc. IFIP Congress. 506--508 . I. E. Sutherland. 1965. The ultimate display. In Proc. IFIP Congress. 506--508."},{"key":"e_1_2_2_56_1","volume-title":"Leiva","author":"Pucihar Klen \u010copi\u010d","year":"2019","unstructured":"Klen \u010copi\u010d Pucihar , Christian Sandor , Matja\u017e Kljun , Wolfgang Huerst , Alexander Plopski , Takafumi Taketomi , Hirokazu Kato , and Luis A . Leiva . 2019 . The Missing Interface: Micro- Gestures on Augmented Objects. In Proc. CHI EA. 1--6. Klen \u010copi\u010d Pucihar, Christian Sandor, Matja\u017e Kljun, Wolfgang Huerst, Alexander Plopski, Takafumi Taketomi, Hirokazu Kato, and Luis A. Leiva. 2019. The Missing Interface: Micro-Gestures on Augmented Objects. In Proc. CHI EA. 1--6."},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.82"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3210240.3210335"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3081333.3081340"},{"key":"e_1_2_2_60_1","doi-asserted-by":"crossref","unstructured":"J. P. Wachs M. K\u00f6lsch H. Stern and Y. Edan. 2011. Vision-based hand-gesture applications. Commun. ACM 54 2 (2011).  J. P. Wachs M. K\u00f6lsch H. Stern and Y. Edan. 2011. Vision-based hand-gesture applications. Commun. ACM 54 2 (2011).","DOI":"10.1145\/1897816.1897838"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984565"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2789168.2790113"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984515"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2004.09.007"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2973750.2973762"},{"key":"e_1_2_2_66_1","volume-title":"Proc. CVPR. 7356--7365","author":"Zhao M.","unstructured":"M. Zhao , T. Li , M. A. Alsheikh , Y. Tian , H. Zhao , A. Torralba , and D. Katabi . 2018. Through-Wall Human Pose Estimation Using Radio Signals . In Proc. CVPR. 7356--7365 . M. Zhao, T. Li, M. A. Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi. 2018. Through-Wall Human Pose Estimation Using Radio Signals. In Proc. CVPR. 7356--7365."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.01021"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/DCOSS.2019.00028"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448110","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3448110","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:59Z","timestamp":1750195499000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448110"}},"subtitle":["Mid-Air Gesture Recognition with Sparse Millimeter-Wave Radar Point Clouds"],"short-title":[],"issued":{"date-parts":[[2021,3,19]]},"references-count":68,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,3,19]]}},"alternative-id":["10.1145\/3448110"],"URL":"https:\/\/doi.org\/10.1145\/3448110","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,19]]},"assertion":[{"value":"2021-03-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}