{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T02:04:40Z","timestamp":1791511480139,"version":"4.3.4"},"reference-count":31,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2020,5,20]],"date-time":"2020-05-20T00:00:00Z","timestamp":1589932800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Existing public domain multi-modal datasets for human action recognition only include actions of interest that have already been segmented from action streams. These datasets cannot be used to study a more realistic action recognition scenario where actions of interest occur randomly and continuously among actions of non-interest or no actions. It is more challenging to recognize actions of interest in continuous action streams since the starts and ends of these actions are not known and need to be determined in an on-the-fly manner. Furthermore, there exists no public domain multi-modal dataset in which video and inertial data are captured simultaneously for continuous action streams. The main objective of this paper is to describe a dataset that is collected and made publicly available, named Continuous Multimodal Human Action Dataset (C-MHAD), in which video and inertial data stream are captured simultaneously in a continuous way. This dataset is then used in an example recognition technique and the results obtained indicate that the fusion of these two sensing modalities increases the F1 scores compared to using each sensing modality individually.<\/jats:p>","DOI":"10.3390\/s20102905","type":"journal-article","created":{"date-parts":[[2020,5,20]],"date-time":"2020-05-20T10:37:38Z","timestamp":1589971058000},"page":"2905","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":36,"title":["C-MHAD: Continuous Multimodal Human Action Dataset of Simultaneous Video and Inertial Sensing"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0666-5551","authenticated-orcid":false,"given":"Haoran","family":"Wei","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4647-6971","authenticated-orcid":false,"given":"Pranav","family":"Chopada","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nasser","family":"Kehtarnavaz","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,5,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1016\/j.patcog.2015.11.019","article-title":"3D Skeleton-based Human Action Classification: A Survey","volume":"53","author":"Presti","year":"2016","journal-title":"Pattern Recognit."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8895","DOI":"10.3390\/s140508895","article-title":"A vision-based system for intelligent monitoring: Human behaviour analysis and privacy by context","volume":"14","author":"Chaaraoui","year":"2014","journal-title":"Sensors"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wei, H., and Kehtarnavaz, N. (2019). Semi-Supervised Faster RCNN-Based Person Detection and Load Classification for Far Field Video Surveillance. Mach. Learn. Knowl. Extr., 1.","DOI":"10.3390\/make1030044"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1662","DOI":"10.1016\/j.eswa.2012.09.004","article-title":"Elderly activities recognition and classification for applications in assisted living","volume":"40","author":"Chernbumroong","year":"2013","journal-title":"Expert Syst. Appl."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1109\/MSP.2015.2502784","article-title":"Radar signal processing for elderly fall detection: The future for in-home monitoring","volume":"33","author":"Amin","year":"2016","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bloom, V., Makris, D., and Argyriou, V. (2012, January 16\u201321). G3D: A gaming action dataset and real time action recognition evaluation framework. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6239175"},{"key":"ref_7","unstructured":"Wang, Y., Yu, T., Shi, L., and Li, Z. (June, January 26). Using human body gestures as inputs for gaming via depth analysis. Proceedings of the IEEE International Conference on Multimedia and Expo, Hannover, Germany."},{"key":"ref_8","unstructured":"Simonyan, K., and Zisserman, A. (2014, January 8\u201313). Two-stream convolutional networks for action recognition in videos. Proceedings of the Advances in Neural Information Processing Systems, Montr\u00e9al, QC, Canada."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 13\u201316). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dawar, N., and Kehtarnavaz, N. (2017, January 17\u201320). Continuous detection and recognition of actions of interest among actions of non-interest using a depth camera. Proceedings of the IEEE International Conference on Image Processing, Beijing, China.","DOI":"10.1109\/ICIP.2017.8297079"},{"key":"ref_11","unstructured":"Chen, C., Kehtarnavaz, N., and Jafari, R. (2014, January 26\u201330). A medication adherence monitoring system for pill bottles based on a wearable inertial sensor. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1109\/MSP.2018.2846286","article-title":"A Survey on Smart Homes for Aging in Place: Toward Solutions to the Specific Needs of the Elderly","volume":"35","author":"Nathan","year":"2018","journal-title":"IEEE Signal. Process. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1109\/TPAMI.2017.2691321","article-title":"Deep Multimodal Feature Analysis for Action Recognition in RGB + D Videos","volume":"40","author":"Shahroudy","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, J., Shahroudy, A., Xu, D., and Wang, G. (2016, January 11\u201314). Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46487-9_50"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Liu, C., Hu, Y., Li, Y., Song, S., and Liu, J. (2017, January 23). PKU-MMD: A large scale benchmark for skeleton-based human action understanding. Proceedings of the Workshop on Visual Analysis in Smart and Connected Communities, Mountain View, CA, USA.","DOI":"10.1145\/3132734.3132739"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Shahroudy, A., Liu, J., Ng, T.T., and Wang, G. (2016, January 27\u201330). Ntu rgb+ d: A large scale dataset for 3D human activity analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.115"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Liu, J., Shahroudy, A., Perez, M.L., Wang, G., Duan, L.Y., and Chichung, A.K. (2019). Ntu rgb+ d 120: A large-scale benchmark for 3D human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2019.2916873"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/THMS.2014.2362520","article-title":"Improving human action recognition using fusion of depth camera and inertial sensors","volume":"45","author":"Chen","year":"2015","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1109\/JSEN.2015.2487358","article-title":"A real-time human action recognition system using depth and inertial sensor fusion","volume":"16","author":"Chen","year":"2016","journal-title":"IEEE Sens. J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4405","DOI":"10.1007\/s11042-015-3177-1","article-title":"A survey of depth and inertial sensor fusion for human action recognition","volume":"76","author":"Chen","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Dawar, N., and Kehtarnavaz, N. (2018, January 12\u201315). A convolutional neural network-based sensor fusion system for monitoring transition movements in healthcare applications. Proceedings of the IEEE 14th International Conference on Control and Automation, Anchorage, AK, USA.","DOI":"10.1109\/ICCA.2018.8444326"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"7019","DOI":"10.1109\/ACCESS.2017.2788558","article-title":"Real-time continuous detection and recognition of subject-specific smart tv gestures via fusion of depth and inertial sensing","volume":"6","author":"Dawar","year":"2018","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Rwigema, J., Choi, H.R., and Kim, T. (2019). A Differential Evolution Approach to Optimize Weights of Dynamic Time Warping for Multi-Sensor Based Gesture Recognition. Sensors, 19.","DOI":"10.3390\/s19051007"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"9660","DOI":"10.1109\/JSEN.2018.2872862","article-title":"Action detection and recognition in continuous action streams by deep learning-based sensing fusion","volume":"18","author":"Dawar","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Wei, H., Jafari, R., and Kehtarnavaz, N. (2019). Fusion of Video and Inertial Sensing for Deep Learning\u2013Based Human Action Recognition. Sensors, 19.","DOI":"10.3390\/s19173680"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"684","DOI":"10.3389\/fneur.2018.00684","article-title":"Thigh-derived inertial sensor metrics to assess the sit-to-stand and stand-to-sit transitions in the timed up and go (TUG) task for quantifying mobility impairment in multiple sclerosis","volume":"9","author":"Witchel","year":"2018","journal-title":"Front. Neurol."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., and Bajcsy, R. (2013, January 15\u201317). Berkeley MHAD: A Comprehensive Multimodal Human Action Database. Proceedings of the IEEE Workshop on Applications on Computer Vision, Tampa, FL, USA.","DOI":"10.1109\/WACV.2013.6474999"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chen, C., Jafari, R., and Kehtarnavaz, N. (2015, January 27\u201330). UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. Proceedings of the 2015 IEEE International Conference on Image Processing, Quebec City, QC, Canada.","DOI":"10.1109\/ICIP.2015.7350781"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Eskofier, B.M., Lee, S.I., Daneault, J.F., Golabchi, F.N., Ferreira-Carvalho, G., Vergara-Diaz, G., Sapienza, S., Costante, G., Klucken, J., and Kautz, T. (2016, January 16\u201320). Recent machine learning advancements in sensor-based mobility analysis: Deep learning for Parkinson\u2019s disease assessment. Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7590787"},{"key":"ref_30","unstructured":"Da Silva, K.B. (2016). How to Use Avidemux: The Open Source Video Editing Sofware for Complete Beginners, CreateSpace Independent Publishing Platform."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"6055","DOI":"10.1109\/JSEN.2020.2973361","article-title":"Simultaneous utilization of inertial and video sensing for action detection and recognition in continuous action streams","volume":"20","author":"Wei","year":"2020","journal-title":"IEEE Sens. J."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/10\/2905\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:30:49Z","timestamp":1760175049000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/10\/2905"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,20]]},"references-count":31,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2020,5]]}},"alternative-id":["s20102905"],"URL":"https:\/\/doi.org\/10.3390\/s20102905","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,20]]}}}