{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T17:07:58Z","timestamp":1783789678488,"version":"3.55.0"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2023,9,27]]},"abstract":"<jats:p>Learning the human--mobility interaction (HMI) on interactive scenes (e.g., how a vehicle turns at an intersection in response to traffic lights and other oncoming vehicles) can enhance the safety, efficiency, and resilience of smart mobility systems (e.g., autonomous vehicles) and many other ubiquitous computing applications. Towards the ubiquitous and understandable HMI learning, this paper considers both \"spoken language\" (e.g., human textual annotations) and \"unspoken language\" (e.g., visual and sensor-based behavioral mobility information related to the HMI scenes) in terms of information modalities from the real-world HMI scenarios. We aim to extract the important but possibly implicit HMI concepts (as the named entities) from the textual annotations (provided by human annotators) through a novel human language and sensor data co-learning design.<\/jats:p>\n          <jats:p>To this end, we propose CG-HMI, a novel Cross-modality Graph fusion approach for extracting important Human-Mobility Interaction concepts from co-learning of textual annotations as well as the visual and behavioral sensor data. In order to fuse both unspoken and spoken \"languages\", we have designed a unified representation called the human--mobility interaction graph (HMIG) for each modality related to the HMI scenes, i.e., textual annotations, visual video frames, and behavioral sensor time-series (e.g., from the on-board or smartphone inertial measurement units). The nodes of the HMIG in these modalities correspond to the textual words (tokenized for ease of processing) related to HMI concepts, the detected traffic participant\/environment categories, and the vehicle maneuver behavior types determined from the behavioral sensor time-series. To extract the inter- and intra-modality semantic correspondences and interactions in the HMIG, we have designed a novel graph interaction fusion approach with differentiable pooling-based graph attention. The resulting graph embeddings are then processed to identify and retrieve the HMI concepts within the annotations, which can benefit the downstream human-computer interaction and ubiquitous computing applications. We have developed and implemented CG-HMI into a system prototype, and performed extensive studies upon three real-world HMI datasets (two on car driving and the third one on e-scooter riding). We have corroborated the excellent performance (on average 13.11% higher accuracy than the other baselines in terms of precision, recall, and F1 measure) and effectiveness of CG-HMI in recognizing and extracting the important HMI concepts through cross-modality learning. Our CG-HMI studies also provide real-world implications (e.g., road safety and driving behaviors) about the interactions between the drivers and other traffic participants.<\/jats:p>","DOI":"10.1145\/3610904","type":"journal-article","created":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T15:45:03Z","timestamp":1695829503000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Cross-Modality Graph-based Language and Sensor Data Co-Learning of Human-Mobility Interaction"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2500-4054","authenticated-orcid":false,"given":"Mahan","family":"Tabatabaie","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, University of Connecticut, Storrs, CT, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1913-6808","authenticated-orcid":false,"given":"Suining","family":"He","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Connecticut, Storrs, CT, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0086-8777","authenticated-orcid":false,"given":"Kang G.","family":"Shin","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Michigan--Ann Arbor, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2022. How Mercedes-Benz Is Using AI & NLP To Give Driving A Tech Makeover. https:\/\/analyticsindiamag.com\/how-mercedes-benz-is-using-ai-nlp-to-give-driving-a-tech-makeover\/."},{"key":"e_1_2_1_2_1","unstructured":"2022. Natural language processing enhances autonomous vehicles experience. https:\/\/www.autonomousvehicleinternational.com\/features\/natural-language-processing-enhances-autonomous-vehicles-experience.html."},{"key":"e_1_2_1_3_1","first-page":"1","article-title":"The City as a Personal Assistant: Turning Urban Landmarks into Conversational Agents for Serving Hyper Local Information","volume":"6","author":"Acer Utku G\u00fcnay","year":"2022","unstructured":"Utku G\u00fcnay Acer, Marc van den Broeck, Chulhong Min, Mallesham Dasari, and Fahim Kawsar. 2022. The City as a Personal Assistant: Turning Urban Landmarks into Conversational Agents for Serving Hyper Local Information. Proc. ACM IMWUT 6, 2 (2022), 1--31.","journal-title":"Proc. ACM IMWUT"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01293"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2021.108421"},{"key":"e_1_2_1_6_1","first-page":"1","article-title":"MobiVQA: Efficient On-Device Visual Question Answering","volume":"6","author":"Cao Qingqing","year":"2022","unstructured":"Qingqing Cao, Prerna Khanna, Nicholas D Lane, and Aruna Balasubramanian. 2022. MobiVQA: Efficient On-Device Visual Question Answering. Proc. ACM IMWUT 6, 2 (2022), 1--23.","journal-title":"Proc. ACM IMWUT"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS47612.2022.9981515"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2742647.2742659"},{"key":"e_1_2_1_9_1","volume-title":"Proc","author":"Chen Dawei","unstructured":"Dawei Chen, Zhixu Li, Binbin Gu, and Zhigang Chen. 2021. Multimodal named entity recognition with image attributes and image knowledge. In Proc. Springer DASFAA. Springer, 186--201."},{"key":"e_1_2_1_10_1","first-page":"1","article-title":"TurnsMap: Enhancing driving safety at intersections with mobile crowdsensing and deep learning","volume":"3","author":"Chen Dongyao","year":"2019","unstructured":"Dongyao Chen and Kang G Shin. 2019. TurnsMap: Enhancing driving safety at intersections with mobile crowdsensing and deep learning. Proc. ACM IMWUT 3, 3 (2019), 1--22.","journal-title":"Proc. ACM IMWUT"},{"key":"e_1_2_1_11_1","volume-title":"Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction. arXiv","author":"Chen Xiang","year":"2022","unstructured":"Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction. arXiv (2022)."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00104"},{"key":"e_1_2_1_13_1","volume-title":"An effective transition-based model for discontinuous NER. arXiv","author":"Dai Xiang","year":"2020","unstructured":"Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020. An effective transition-based model for discontinuous NER. arXiv (2020)."},{"key":"e_1_2_1_14_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv (2018)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2022.08.011"},{"key":"e_1_2_1_16_1","volume-title":"Retrieved","year":"2022","unstructured":"Drover. 2022. AI-Powered Computer Vision for Micromobility. Retrieved October 11, 2022 from https:\/\/drover.ai\/"},{"key":"e_1_2_1_17_1","volume-title":"DADA: Driver attention prediction in driving accident scenarios","author":"Fang Jianwu","year":"2021","unstructured":"Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, and Hongkai Yu. 2021. DADA: Driver attention prediction in driving accident scenarios. IEEE T-ITS (2021)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01500"},{"key":"e_1_2_1_19_1","volume-title":"Shin","author":"He Suining","year":"2022","unstructured":"Suining He and Kang G. Shin. 2022. Socially-Equitable Interactive Graph Information Fusion-Based Prediction for Urban Dockless E-Scooter Sharing. In Proc. WWW. Association for Computing Machinery, 3269--3279."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.440"},{"key":"e_1_2_1_21_1","volume-title":"Bidirectional LSTM-CRF models for sequence tagging. arXiv","author":"Huang Zhiheng","year":"2015","unstructured":"Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv (2015)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01084"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00968"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01216-8_35"},{"key":"e_1_2_1_25_1","volume-title":"Semi-supervised classification with graph convolutional networks. arXiv","author":"Kipf Thomas N","year":"2016","unstructured":"Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv (2016)."},{"key":"e_1_2_1_26_1","volume-title":"A survey on imitation learning techniques for end-to-end autonomous vehicles","author":"Mero Luc Le","year":"2022","unstructured":"Luc Le Mero, Dewei Yi, Mehrdad Dianati, and Alexandros Mouzakitis. 2022. A survey on imitation learning techniques for end-to-end autonomous vehicles. IEEE T-ITS (2022)."},{"key":"e_1_2_1_27_1","volume-title":"A span-based model for joint overlapped and discontinuous named entity recognition. arXiv","author":"Li Fei","year":"2021","unstructured":"Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong Ji. 2021. A span-based model for joint overlapped and discontinuous named entity recognition. arXiv (2021)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449995"},{"key":"e_1_2_1_29_1","first-page":"50","article-title":"A survey on deep learning for named entity recognition","volume":"34","author":"Li Jing","year":"2020","unstructured":"Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE TKDE 34, 1 (2020), 50--70.","journal-title":"IEEE TKDE"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00298"},{"key":"e_1_2_1_31_1","volume-title":"A deep learning approach to detect real-time vehicle maneuvers based on smartphone sensors","author":"Li Pei","year":"2020","unstructured":"Pei Li, Mohamed Abdel-Aty, Qing Cai, and Zubayer Islam. 2020. A deep learning approach to detect real-time vehicle maneuvers based on smartphone sensors. IEEE T-ITS (2020)."},{"key":"e_1_2_1_32_1","volume-title":"IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text. arXiv preprint arXiv:2210.14395","author":"Moon Seungwhan","year":"2022","unstructured":"Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Alireza Dirafzoon, Aparajita Saraf, Amy Bearman, and Babak Damavandi. 2022. IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text. arXiv preprint arXiv:2210.14395 (2022)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2019.8803392"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.124"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00051"},{"key":"e_1_2_1_36_1","first-page":"1","article-title":"Multimodal deep learning for activity and context recognition","volume":"1","author":"Radu Valentin","year":"2018","unstructured":"Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Mahesh K Marina, and Fahim Kawsar. 2018. Multimodal deep learning for activity and context recognition. Proc. ACM IMWUT 1, 4 (2018), 1--27.","journal-title":"Proc. ACM IMWUT"},{"key":"e_1_2_1_37_1","first-page":"26","article-title":"The Web of Things","volume":"48","author":"Raggett Dave","year":"2015","unstructured":"Dave Raggett. 2015. The Web of Things: Challenges and Opportunities. Computer 48, 5 (2015), 26--32.","journal-title":"Challenges and Opportunities. Computer"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.91"},{"key":"e_1_2_1_39_1","volume-title":"Leonardo Taccari, Samuele Salti, Luca Kubin, Fabio Schoen, and Francesco Sambo.","author":"Simoncini Matteo","year":"2022","unstructured":"Matteo Simoncini, Douglas Coimbra de Andrade, Leonardo Taccari, Samuele Salti, Luca Kubin, Fabio Schoen, and Francesco Sambo. 2022. Unsafe Maneuver Classification From Dashcam Video and GPS\/IMU Sensors Using Spatio-Temporal Attention Selector. IEEE T-ITS (2022)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-86534-4_11"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i15.17633"},{"key":"e_1_2_1_42_1","volume-title":"Proc. ACM IMWUT 6, 4, Article 205 (Jan","author":"Tabatabaie Mahan","year":"2023","unstructured":"Mahan Tabatabaie and Suining He. 2023. Naturalistic E-Scooter Maneuver Recognition with Federated Contrastive Rider Interaction Learning. Proc. ACM IMWUT 6, 4, Article 205 (Jan 2023), 27 pages."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474717.3483911"},{"key":"e_1_2_1_44_1","first-page":"1","article-title":"Driver Maneuver Identification with Multi-Representation Learning and Meta Model Update Designs","volume":"6","author":"Tabatabaie Mahan","year":"2022","unstructured":"Mahan Tabatabaie, Suining He, and Xi Yang. 2022. Driver Maneuver Identification with Multi-Representation Learning and Meta Model Update Designs. Proc. ACM IMWUT 6, 2 (2022), 1--23.","journal-title":"Proc. ACM IMWUT"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6434"},{"key":"e_1_2_1_46_1","unstructured":"Ultralytics. 2022. Yolov5. Retrieved October 3 2022 from https:\/\/github.com\/ultralytics\/yolov5"},{"key":"e_1_2_1_47_1","first-page":"1","article-title":"Context recognition in-the-wild: Unified model for multi-modal sensors and multi-label classification","volume":"1","author":"Vaizman Yonatan","year":"2018","unstructured":"Yonatan Vaizman, Nadir Weibel, and Gert Lanckriet. 2018. Context recognition in-the-wild: Unified model for multi-modal sensors and multi-label classification. Proc. ACM IMWUT 1, 4 (2018), 1--22.","journal-title":"Proc. ACM IMWUT"},{"key":"e_1_2_1_48_1","volume-title":"Graph attention networks. arXiv preprint arXiv:1710.10903","author":"Veli\u010dkovi\u0107 Petar","year":"2017","unstructured":"Petar Veli\u010dkovi\u0107, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)."},{"key":"e_1_2_1_49_1","volume-title":"International Conference on Database Systems for Advanced Applications","author":"Wang Xuwu","unstructured":"Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Jiabo Ye, Ming Yan, and Yanghua Xiao. 2022. PromptMNER:Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition. In International Conference on Database Systems for Advanced Applications. Springer, 297--305."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372278.3390713"},{"key":"e_1_2_1_51_1","volume-title":"Proc. ICML. PMLR, 6861--6871","author":"Wu Felix","year":"2019","unstructured":"Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019. Simplifying graph convolutional networks. In Proc. ICML. PMLR, 6861--6871."},{"key":"e_1_2_1_52_1","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV45572.2020.9093524"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the Thirteenth Language Resources and Evaluation Conference. 6802--6807","author":"Xu Shuo","year":"2022","unstructured":"Shuo Xu, Yuxiang Jia, Changyong Niu, and Hongying Zan. 2022. MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. 6802--6807."},{"key":"e_1_2_1_55_1","volume-title":"TENER: adapting transformer encoder for named entity recognition. arXiv preprint arXiv:1911.04474","author":"Yan Hang","year":"2019","unstructured":"Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. TENER: adapting transformer encoder for named entity recognition. arXiv preprint arXiv:1911.04474 (2019)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1174"},{"key":"e_1_2_1_57_1","volume-title":"Proc. NeurIPS 31","author":"Ying Zhitao","year":"2018","unstructured":"Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018. Hierarchical graph representation learning with differentiable pooling. Proc. NeurIPS 31 (2018)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00271"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.306"},{"key":"e_1_2_1_60_1","doi-asserted-by":"crossref","unstructured":"Yuting Zhan and Hamed Haddadi. 2019. Towards automating smart homes: Contextual and temporal dynamics of activity prediction. In UbiComp\/ISWC. 413--417.","DOI":"10.1145\/3341162.3349307"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17687"},{"key":"e_1_2_1_62_1","first-page":"1","article-title":"Adaptive Co-attention Network for Named Entity Recognition in Tweets","volume":"32","author":"Zhang Qi","year":"2018","unstructured":"Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive Co-attention Network for Named Entity Recognition in Tweets. Proc. AAAI 32, 1 (Apr. 2018).","journal-title":"Proc. AAAI"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610904","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610904","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T16:29:25Z","timestamp":1753720165000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610904"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":62,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9,27]]}},"alternative-id":["10.1145\/3610904"],"URL":"https:\/\/doi.org\/10.1145\/3610904","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,27]]},"assertion":[{"value":"2023-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}