{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T05:17:15Z","timestamp":1790831835296,"version":"4.1.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T00:00:00Z","timestamp":1709683200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Institute On Aging of the National Institutes of Health","award":["P30AG073105"],"award-info":[{"award-number":["P30AG073105"]}]},{"DOI":"10.13039\/501100006374","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1901048, CNS-1925767, CNS-2128588"],"award-info":[{"award-number":["CNS-1901048, CNS-1925767, CNS-2128588"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Samsung collaboration grant"},{"name":"Google Ph.D. Fellowship"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2024,3,6]]},"abstract":"<jats:p>Egocentric non-intrusive sensing of human activities of daily living (ADL) in free-living environments represents a holy grail in ubiquitous computing. Existing approaches, such as egocentric vision and wearable motion sensors, either can be intrusive or have limitations in capturing non-ambulatory actions. To address these challenges, we propose EgoADL, the first egocentric ADL sensing system that uses an in-pocket smartphone as a multi-modal sensor hub to capture body motion, interactions with the physical environment and daily objects using non-visual sensors (audio, wireless sensing, and motion sensors). We collected a 120-hour multimodal dataset and annotated 20-hour data into 221 ADL, 70 object interactions, and 91 actions. EgoADL proposes multi-modal frame-wise slow-fast encoders to learn the feature representation of multi-sensory data that characterizes the complementary advantages of different modalities and adapt a transformer-based sequence-to-sequence model to decode the time-series sensor signals into a sequence of words that represent ADL. In addition, we introduce a self-supervised learning framework that extracts intrinsic supervisory signals from the multi-modal sensing data to overcome the lack of labeling data and achieve better generalization and extensibility. Our experiments in free-living environments demonstrate that EgoADL can achieve comparable performance with video-based approaches, bringing the vision of ambient intelligence closer to reality.<\/jats:p>","DOI":"10.1145\/3643553","type":"journal-article","created":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T13:12:36Z","timestamp":1709730756000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Multimodal Daily-Life Logging in Free-living Environment Using Non-Visual Egocentric Sensors on a Smartphone"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9563-1398","authenticated-orcid":false,"given":"Ke","family":"Sun","sequence":"first","affiliation":[{"name":"University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1777-3587","authenticated-orcid":false,"given":"Chunyu","family":"Xia","sequence":"additional","affiliation":[{"name":"University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9688-8056","authenticated-orcid":false,"given":"Xinyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of California, San Diego, La Jolla, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0814-9144","authenticated-orcid":false,"given":"Hao","family":"Chen","sequence":"additional","affiliation":[{"name":"Standard and Mobility Innovation Lab, Samsung Research America, Plano, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9775-2783","authenticated-orcid":false,"given":"Charlie Jianzhong","family":"Zhang","sequence":"additional","affiliation":[{"name":"Standard and Mobility Innovation Lab, Samsung Research America, Plano, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,3,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Nature","author":"Haque Albert","year":"2020","unstructured":"Albert Haque, Arnold Milstein, and Li Fei-Fei. Illuminating the dark spaces of healthcare with ambient intelligence. Nature, 2020."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.721"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of ECCV","author":"Damen Dima","year":"2018","unstructured":"Dima Damen, Hazel Doughty, and et.al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of ECCV, 2018."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00772"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01842"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00111"},{"key":"e_1_2_1_7_1","volume-title":"ACM Computing Surveys (CSUR)","author":"Chen Kaixuan","year":"2021","unstructured":"Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. ACM Computing Surveys (CSUR), 2021."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of IEEE\/CVF ICCV","author":"Kazakos Evangelos","year":"2019","unstructured":"Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of IEEE\/CVF ICCV, 2019."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00548"},{"key":"e_1_2_1_10_1","volume-title":"Towards continual egocentric activity recognition: A multi-modal egocentric activity dataset for continual learning. arXiv preprint arXiv:2301.10931","author":"Xu Linfeng","year":"2023","unstructured":"Linfeng Xu, Qingbo Wu, Lili Pan, Fanman Meng, Hongliang Li, Chiyuan He, Hanxin Wang, Shaoxu Cheng, and Yu Dai. Towards continual egocentric activity recognition: A multi-modal egocentric activity dataset for continual learning. arXiv preprint arXiv:2301.10931, 2023."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475557"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/159544.159617"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46448-0_31"},{"key":"e_1_2_1_14_1","volume-title":"Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626","author":"Sigurdsson Gunnar A","year":"2018","unstructured":"Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018."},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"EPIC-KITCHENS-100- 2021 Challenges Report 2022. https:\/\/epic-kitchens.github.io\/Reports\/EPIC-KITCHENS-Challenges-2021-Report.pdf.","DOI":"10.4324\/9781003108559-13"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2022.3183112"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of IEEE ICASSP","author":"Gemmeke Jort F","year":"2017","unstructured":"Jort F Gemmeke, Daniel PW Ellis, and et.al. Audio set: An ontology and human-labeled dataset for audio events. In Proceedings of IEEE ICASSP, 2017."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of ACM MobiCom","author":"Yan","year":"2014","unstructured":"Yan Wang and et.al. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of ACM MobiCom, 2014."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2789168.2790093"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3241539.3241548"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372224.3380900"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58536-5_7"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of WWW","author":"Shuochao","year":"2017","unstructured":"Shuochao Yao and et.al. Deepsense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of WWW, 2017."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485730.3485937"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.33682\/006b-jx26"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of ACM UIST","author":"Laput Gierad","year":"2018","unstructured":"Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. Ubicoustics: Plug-and-play acoustic activity recognition. In Proceedings of ACM UIST, 2018."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of ACM MobiCom","author":"Xiaomin","year":"2022","unstructured":"Xiaomin Ouyang and et.al. Cosmo: contrastive fusion learning with small data for multimodal human activity recognition. In Proceedings of ACM MobiCom, 2022."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534582"},{"key":"e_1_2_1_29_1","volume-title":"IEEE TPAMI","author":"Li Yin","year":"2021","unstructured":"Yin Li, Miao Liu, and Jame Rehg. In the eye of the beholder: Gaze and actions in first person video. IEEE TPAMI, 2021."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161190"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594368.2594392"},{"key":"e_1_2_1_32_1","volume-title":"Panns: Large-scale pretrained audio neural networks for audio pattern recognition","author":"Kong Qiuqiang","year":"2020","unstructured":"Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE\/ACM TASLP, 2020."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of ACM CHI","author":"Jason","year":"2020","unstructured":"Jason Wu and et.al. Automated class discovery and one-shot interactions for acoustic activity recognition. In Proceedings of ACM CHI, 2020."},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of ACM IMWUT","author":"Adaimi Rebecca","year":"2021","unstructured":"Rebecca Adaimi, Howard Yong, and Edison Thomaz. Ok google, what am i doing? acoustic activity recognition bounded by conversational assistant interactions. Proceedings of ACM IMWUT, 2021."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of ACM MobiCom","author":"Wang Yan","year":"2014","unstructured":"Yan Wang, Jian Liu, and et.al. E-eyes: In-home device-free activity identification using fine-grained WiFi signatures. In Proceedings of ACM MobiCom, 2014."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3463506"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of ACM IMWUT","author":"Ian Chi","year":"2021","unstructured":"Chi Ian Tang and et.al. Selfhar: Improving human activity recognition through self-training with unlabeled data. Proceedings of ACM IMWUT, 2021."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448074"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of USENIX NSDI","author":"Chuhan","year":"2018","unstructured":"Chuhan Gao and et.al. Livetag: Sensing human-object interaction through passive chipless wifi tags. In Proceedings of USENIX NSDI, 2018."},{"key":"e_1_2_1_40_1","volume-title":"IEEE TPAMI","author":"Jing Longlong","year":"2020","unstructured":"Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE TPAMI, 2020."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550299"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of NeurIPS","author":"Alwassel Humam","year":"2020","unstructured":"Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-supervised learning by cross-modal audio-video clustering. Proceedings of NeurIPS, 2020."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of NeurIPS","author":"Hassan","year":"2021","unstructured":"Hassan Akbari and et.al. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Proceedings of NeurIPS, 2021."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00791"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of ACM WiNTECH","author":"Francesco","year":"2019","unstructured":"Francesco Gringoli and et.al. Free your csi: A channel state information extraction platform for modern wi-fi chipsets. In Proceedings of ACM WiNTECH, 2019."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3131473.3131476"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300568"},{"key":"e_1_2_1_48_1","volume":"201","author":"Wu Dan","unstructured":"Dan Wu and et.al. Device-free wifi human sensing: From pattern-based to model-based approaches. IEEE Communications Magazine, 2017.","journal-title":"IEEE Communications Magazine"},{"key":"e_1_2_1_49_1","volume-title":"Dezhi Hong, Rajesh K Gupta, and Jingbo Shang. Modeling label semantics improves activity recognition. arXiv preprint arXiv:2301.03462","author":"Zhang Xiyuan","year":"2023","unstructured":"Xiyuan Zhang, Ranak Roy Chowdhury, Dezhi Hong, Rajesh K Gupta, and Jingbo Shang. Modeling label semantics improves activity recognition. arXiv preprint arXiv:2301.03462, 2023."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2494091.2496039"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313426"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58548-8_13"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00630"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP39728.2021.9413376"},{"key":"e_1_2_1_55_1","volume-title":"An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929","author":"Dosovitskiy Alexey","year":"2020","unstructured":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2021-698"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of AAAI","author":"Ashwin","year":"2016","unstructured":"Ashwin K Vijayakumar and et.al. Diverse beam search: Decoding diverse solutions from neural sequence models. In Proceedings of AAAI, 2016."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01264-9_9"},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of NAACL-HLT","author":"Jacob","year":"2019","unstructured":"Jacob Devlin and et.al. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 2019."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01228-1_38"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.2991965"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00608"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of IEEE\/CVF CVPR","author":"Yanghao","year":"2021","unstructured":"Yanghao Li and et.al. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of IEEE\/CVF CVPR, 2021."},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of IEEE PIMRC","author":"Vasanth Swaminathan","year":"2014","unstructured":"Swaminathan Vasanth Rajaraman and et.al. Energy consumption anatomy of live video streaming from a smartphone. In Proceedings of IEEE PIMRC, 2014."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of IEEE CVPR","author":"Mark","year":"2018","unstructured":"Mark Sandler and et.al. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of IEEE CVPR, 2018."},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of ACM SenSys","author":"Jianwei","year":"2018","unstructured":"Jianwei Qian and et.al. Hidebehind: Enjoy voice input with voiceprint unclonability and anonymity. In Proceedings of ACM SenSys, 2018."},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of ACM MobiCom","author":"Wang Wei","year":"2016","unstructured":"Wei Wang, Alex X. Liu, and Ke Sun. Device-free gesture tracking using acoustic signals. In Proceedings of ACM MobiCom, 2016."},{"key":"e_1_2_1_68_1","volume-title":"Proceedings of ACM WiSec","author":"Anand S Abhishek","year":"2021","unstructured":"S Abhishek Anand, Chen Wang, Jian Liu, Nitesh Saxena, and Yingying Chen. Spearphone: a lightweight speech privacy exploit via accelerometer-sensed reverberations from smartphone loudspeakers. In Proceedings of ACM WiSec, 2021."},{"key":"e_1_2_1_69_1","unstructured":"Angularjs typeahead 2022. https:\/\/www.npmjs.com\/package\/ngx-typeahead."},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of NeurIPS","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, 2017."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643553","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643553","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T13:04:45Z","timestamp":1755867885000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643553"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,6]]},"references-count":70,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,6]]}},"alternative-id":["10.1145\/3643553"],"URL":"https:\/\/doi.org\/10.1145\/3643553","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,6]]},"assertion":[{"value":"2024-03-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}