{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T04:46:40Z","timestamp":1773463600562,"version":"3.50.1"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"3s","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["72172057, 92046026, 71701089"],"award-info":[{"award-number":["72172057, 92046026, 71701089"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Natural Science Foundation of the Higher Education Institutions of Jiangsu Province, China","award":["21KJB520034"],"award-info":[{"award-number":["21KJB520034"]}]},{"name":"Fundamental Research on Advanced Leading Technology Project of Jiangsu Province","award":["BK20192004C, BK20202011"],"award-info":[{"award-number":["BK20192004C, BK20202011"]}]},{"DOI":"10.13039\/501100013058","name":"Jiangsu Provincial Key Research and Development Program","doi-asserted-by":"crossref","award":["BE2020001-3"],"award-info":[{"award-number":["BE2020001-3"]}],"id":[{"id":"10.13039\/501100013058","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100005272","name":"National Center for International Joint Research on E-Business Information Processing","doi-asserted-by":"crossref","award":["2013B01035"],"award-info":[{"award-number":["2013B01035"]}],"id":[{"id":"10.13039\/501100005272","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:p>\n            This paper explores human activities recognition from sensor-based multi-dimensional streams. Recently, deep learning-based methods such as LSTM and CNN have achieved important progress in practical application scenarios. However, in most previous deep learning-based methods exist potential challenges such as class imbalance and multi-modal heterogeneity with time and sensor signals. To handle those problems, we propose a\n            <jats:bold>graph LSTM and Metric Learning model (GLML)<\/jats:bold>\n            with multiple construction graph fusion by modeling the sensor-aspect signals and the graph-aspect activities. GLML is a semi-supervised co-training architecture, which can be seen as several iteratively pseudo-labels sampling processing in the unlabeled data. Specifically, we construct three graphs to capture the different relations in each timestamp. Meanwhile, the graph attention model and attention mechanism are proposed to integrate multiple graph interactions for different sensor signals. Furthermore, to obtain a fixed representation of hidden state units and their neighboring nodes, we introduce the Graph LSTM to learn the graph-aspect relations from graph-structured constructed graphs. Notably, we propose a multi-task classification model combining loss function for classification distribution with deep metric learning to enhance the representation ability of the multi-modal sensor data. Experimental results on three public datasets demonstrate that our proposed GLML model has at least 2.44% improved in average against the state-of-the-art methods.\n          <\/jats:p>","DOI":"10.1145\/3561387","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T12:21:39Z","timestamp":1662639699000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Sensor-based Human Activity Recognition Using Graph LSTM and Multi-task Classification Model"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9942-3243","authenticated-orcid":false,"given":"Jie","family":"Cao","sequence":"first","affiliation":[{"name":"School of Management, Hefei University of Technology, Anhui, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4726-7493","authenticated-orcid":false,"given":"Youquan","family":"Wang","sequence":"additional","affiliation":[{"name":"Jiangsu Provincial Key Laboratory of E-Business, Nanjing University of Finance and Economics, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1286-2578","authenticated-orcid":false,"given":"Haicheng","family":"Tao","sequence":"additional","affiliation":[{"name":"Jiangsu Provincial Key Laboratory of E-Business, Nanjing University of Finance and Economics, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7047-2690","authenticated-orcid":false,"given":"Xiang","family":"Guo","sequence":"additional","affiliation":[{"name":"School of Computer Science, Wuhan University, Hubei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,10,31]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10044-016-0570-y"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/1922649.1922653"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.3390\/s22041373"},{"key":"e_1_3_2_5_2","article-title":"Deep activity recognition models with triaxial accelerometers","author":"Alsheikh Mohammad Abu","year":"2015","unstructured":"Mohammad Abu Alsheikh, Ahmed Selim, Dusit Niyato, Linda Doyle, Shaowei Lin, and Hwee-Pink Tan. 2015. Deep activity recognition models with triaxial accelerometers. arXiv preprint arXiv:1511.04664 (2015).","journal-title":"arXiv preprint arXiv:1511.04664"},{"key":"e_1_3_2_6_2","first-page":"3","volume-title":"Esann","author":"Anguita Davide","year":"2013","unstructured":"Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. 2013. A public domain dataset for human activity recognition using smartphones. In Esann, Vol. 3. 3."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.3390\/s20092702"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397323"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341162.3349335"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2010.5665869"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2499621"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2012.12.014"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/186"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2927224"},{"key":"e_1_3_2_15_2","article-title":"Deep learning for sensor-based human activity recognition: Overview, challenges and opportunities","author":"Chen Kaixuan","year":"2020","unstructured":"Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2020. Deep learning for sensor-based human activity recognition: Overview, challenges and opportunities. arXiv preprint arXiv:2001.07416 (2020).","journal-title":"arXiv preprint arXiv:2001.07416"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2012.2198883"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3441628"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2599174"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472290"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3090076"},{"key":"e_1_3_2_21_2","first-page":"2535","volume-title":"International Conference on Machine Learning","author":"Hacohen Guy","year":"2019","unstructured":"Guy Hacohen and Daphna Weinshall. 2019. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning. PMLR, 2535\u20132544."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.138"},{"issue":"4","key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3409332","article-title":"Knowledge-driven egocentric multimodal activity recognition","volume":"16","author":"Huang Yi","year":"2020","unstructured":"Yi Huang, Xiaoshan Yang, Junyu Gao, Jitao Sang, and Changsheng Xu. 2020. Knowledge-driven egocentric multimodal activity recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16, 4 (2020), 1\u2013133.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00521"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2021.04.062"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-017-0495-0"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/SURV.2012.110112.00192"},{"issue":"6","key":"e_1_3_2_28_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3448021","article-title":"Similarity embedding networks for robust human activity recognition","volume":"15","author":"Li Chenglin","year":"2021","unstructured":"Chenglin Li, Carrie Lu Tong, Di Niu, Bei Jiang, Xiao Zuo, Lei Cheng, Jian Xiong, and Jianming Yang. 2021. Similarity embedding networks for robust human activity recognition. ACM Transactions on Knowledge Discovery from Data (TKDD) 15, 6 (2021), 1\u201317.","journal-title":"ACM Transactions on Knowledge Discovery from Data (TKDD)"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46448-0_8"},{"key":"e_1_3_2_30_2","first-page":"4981","volume-title":"Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence","author":"Liu Fanzhen","year":"2021","unstructured":"Fanzhen Liu, Shan Xue, Jia Wu, Chuan Zhou, Wenbin Hu, Cecile Paris, Surya Nepal, Jian Yang, and Philip S. Yu. 2021. Deep learning for community detection: Progress, challenges and opportunities. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 4981\u20134987."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3365212"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2016.07.024"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3380999"},{"issue":"57","key":"e_1_3_2_34_2","first-page":"1","article-title":"Self-paced multi-view co-training.","volume":"21","author":"Ma Fan","year":"2020","unstructured":"Fan Ma, Deyu Meng, Xuanyi Dong, and Yi Yang. 2020. Self-paced multi-view co-training.Journal of Machine Learning Research 21, 57 (2020), 1\u201338.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_35_2","first-page":"3109","volume-title":"IJCAI","author":"Ma Haojie","year":"2019","unstructured":"Haojie Ma, Wenzhong Li, Xiao Zhang, Songcheng Gao, and Sanglu Lu. 2019. AttnSense: Multi-level attention mechanism for multimodal human activity recognition. In IJCAI. 3109\u20133115."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2021.3118815"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377882"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2018.03.056"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.3390\/s16010115"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2020.103429"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i13.17416"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2012.13"},{"key":"e_1_3_2_43_2","first-page":"1","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR 2021)","author":"Shang Chao","year":"2021","unstructured":"Chao Shang, Jie Chen, and Jinbo Bi. 2021. Discrete graph structure learning for forecasting multiple time series. In Proceedings of the International Conference on Learning Representations (ICLR 2021). 1\u201313."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01228-1_19"},{"key":"e_1_3_2_45_2","article-title":"Spatiotemporal co-attention recurrent neural networks for human-skeleton motion prediction","author":"Shu Xiangbo","year":"2019","unstructured":"Xiangbo Shu, Liyan Zhang, Guo-Jun Qi, Wei Liu, and Jinhui Tang. 2019. Spatiotemporal co-attention recurrent neural networks for human-skeleton motion prediction. arXiv preprint arXiv:1909.13245 (2019).","journal-title":"arXiv preprint arXiv:1909.13245"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2009.24"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3137396"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/PerCom53586.2022.9762387"},{"key":"e_1_3_2_49_2","article-title":"Coherence constrained graph LSTM for group activity recognition","author":"Tang Jinhui","year":"2019","unstructured":"Jinhui Tang, Xiangbo Shu, Rui Yan, and Liyan Zhang. 2019. Coherence constrained graph LSTM for group activity recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_2_50_2","first-page":"1","volume-title":"Proceedings of The International Conference on Learning Representations (ICLR 2018)","author":"Veli\u010dkovi\u0107 Petar","year":"2018","unstructured":"Petar Veli\u010dkovi\u0107, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of The International Conference on Learning Representations (ICLR 2018). 1\u201312."},{"key":"e_1_3_2_51_2","article-title":"Temporal relational modeling with self-supervision for action segmentation","author":"Wang Dong","year":"2020","unstructured":"Dong Wang, Di Hu, Xingjian Li, and Dejing Dou. 2020. Temporal relational modeling with self-supervision for action segmentation. arXiv preprint arXiv:2012.07508 (2020).","journal-title":"arXiv preprint arXiv:2012.07508"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.387"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2018.02.010"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6907"},{"key":"e_1_3_2_55_2","volume-title":"Unsupervised Learning in Human Activity Recognition: A First Foray into Clustering Data Gathered from Wearable Sensors","author":"Weber Niklas","year":"2017","unstructured":"Niklas Weber, Aki H\u00e4rm\u00e4, and Eindhoven Prof. Dr. Tom Heskes. 2017. Unsupervised Learning in Human Activity Recognition: A First Foray into Clustering Data Gathered from Wearable Sensors. Ph.D. Dissertation. Ph.D. Thesis, Radboud University, Nijmegen, The Netherlands."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3380980"},{"key":"e_1_3_2_57_2","first-page":"3995","volume-title":"Twenty-fourth International Joint Conference on Artificial Intelligence","author":"Yang Jianbo","year":"2015","unstructured":"Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. 2015. Deep convolutional neural networks on multichannel time series for human activity recognition. In Twenty-fourth International Joint Conference on Artificial Intelligence. 3995\u20134001."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052577"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2017.8257967"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/2110363.2110433"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.5555\/3304889.3305093"},{"key":"e_1_3_2_62_2","article-title":"Facial depression recognition by deep joint label distribution and metric learning","author":"Zhou Xiuzhuang","year":"2020","unstructured":"Xiuzhuang Zhou, Zeqiang Wei, Min Xu, Shan Qu, and Guodong Guo. 2020. Facial depression recognition by deep joint label distribution and metric learning. IEEE Transactions on Affective Computing (2020).","journal-title":"IEEE Transactions on Affective Computing"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3561387","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3561387","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:35Z","timestamp":1750186835000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3561387"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":61,"journal-issue":{"issue":"3s","published-print":{"date-parts":[[2022,10,31]]}},"alternative-id":["10.1145\/3561387"],"URL":"https:\/\/doi.org\/10.1145\/3561387","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,31]]},"assertion":[{"value":"2021-11-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-08-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-10-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}