{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T03:33:05Z","timestamp":1792553585037,"version":"4.5.0"},"reference-count":36,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2019,8,24]],"date-time":"2019-08-24T00:00:00Z","timestamp":1566604800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents the simultaneous utilization of video images and inertial signals that are captured at the same time via a video camera and a wearable inertial sensor within a fusion framework in order to achieve a more robust human action recognition compared to the situations when each sensing modality is used individually. The data captured by these sensors are turned into 3D video images and 2D inertial images that are then fed as inputs into a 3D convolutional neural network and a 2D convolutional neural network, respectively, for recognizing actions. Two types of fusion are considered\u2014Decision-level fusion and feature-level fusion. Experiments are conducted using the publicly available dataset UTD-MHAD in which simultaneous video images and inertial signals are captured for a total of 27 actions. The results obtained indicate that both the decision-level and feature-level fusion approaches generate higher recognition accuracies compared to the approaches when each sensing modality is used individually. The highest accuracy of 95.6% is obtained for the decision-level fusion approach.<\/jats:p>","DOI":"10.3390\/s19173680","type":"journal-article","created":{"date-parts":[[2019,8,26]],"date-time":"2019-08-26T04:38:23Z","timestamp":1566794303000},"page":"3680","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":64,"title":["Fusion of Video and Inertial Sensing for Deep Learning\u2013Based Human Action Recognition"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0666-5551","authenticated-orcid":false,"given":"Haoran","family":"Wei","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roozbeh","family":"Jafari","sequence":"additional","affiliation":[{"name":"Center for Remote Health Technologies and System, Texas A&amp;M University, College Station, TX 77843, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nasser","family":"Kehtarnavaz","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,8,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1016\/j.patcog.2015.11.019","article-title":"3D Skeleton-based Human Action Classification: A Survey","volume":"53","author":"Presti","year":"2016","journal-title":"Pattern Recognit."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Dawar, N., and Kehtarnavaz, N. (2017, January 17\u201320). Continuous detection and recognition of actions of interest among actions of non-interest using a depth camera. Proceedings of the IEEE International Conference on Image Processing, Beijing, China.","DOI":"10.1109\/ICIP.2017.8297079"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"5197","DOI":"10.3390\/s150305197","article-title":"Continuous human action recognition using depth-MHI-HOG and a spotter model","volume":"15","author":"Eum","year":"2015","journal-title":"Sensors"},{"key":"ref_4","unstructured":"Chu, X., Ouyang, W., Li, H., and Wang, X. (July, January 26). Structured feature learning for pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"8895","DOI":"10.3390\/s140508895","article-title":"A vision-based system for intelligent monitoring: Human behaviour analysis and privacy by context","volume":"14","author":"Chaaraoui","year":"2014","journal-title":"Sensors"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wei, H., Laszewski, M., and Kehtarnavaz, N. (2018, January 2\u201312). Deep Learning-Based Person Detection and Classification for Far Field Video Surveillance. Proceedings of the 13th IEEE Dallas Circuits and Systems Conference, Dallas, TX, USA.","DOI":"10.1109\/DCAS.2018.8620111"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2329","DOI":"10.1016\/j.patcog.2015.03.006","article-title":"Semantic human activity recognition: A literature review","volume":"48","author":"Ziaeefard","year":"2015","journal-title":"Pattern Recognit."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"756","DOI":"10.3390\/make1030044","article-title":"Semi-Supervised Faster RCNN-Based Person Detection and Load Classification for Far Field Video Surveillance","volume":"1","author":"Wei","year":"2019","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Van Gemert, J.C., Jain, M., Gati, E., and Snoek, C.G. (2015, January 7\u201310). APT: Action localization proposals from dense trajectories. Proceedings of the British Machine Vision Conference 2015: BMVC 2015, Swansea, UK.","DOI":"10.5244\/C.29.177"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhu, H., Vial, R., and Lu, S. (2017, January 22\u201329). Tornado: A spatio-temporal convolutional regression network for video action proposal. Proceedings of the CVPR, Venice, Italy.","DOI":"10.1109\/ICCV.2017.619"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bloom, V., Makris, D., and Argyriou, V. (2012, January 16\u201321). G3D: A gaming action dataset and real time action recognition evaluation framework. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6239175"},{"key":"ref_12","unstructured":"Wang, Y., Yu, T., Shi, L., and Li, Z. (June, January 26). Using human body gestures as inputs for gaming via depth analysis. Proceedings of the IEEE International Conference on Multimedia and Expo, Hannover, Germany."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wang, L., Zang, J., Zhang, Q., Niu, Z., Hua, G., and Zheng, N. (2018). Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network. Sensors, 7.","DOI":"10.3390\/s18071979"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, H., and Schmid, C. (2013, January 3\u20136). Action recognition with improved trajectories. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.","DOI":"10.1109\/ICCV.2013.441"},{"key":"ref_15","unstructured":"Simonyan, K., and Zisserman, A. (2014, January 8\u201313). Two-stream convolutional networks for action recognition in videos. Proceedings of the Advances in Neural Information Processing Systems, Montr\u00e9al, QC, Canada."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 13\u201316). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Avil\u00e9s-Cruz, C., Ferreyra-Ram\u00edrez, A., Z\u00fa\u00f1iga-L\u00f3pez, A., and Villegas-Cort\u00e9z, J. (2019). Coarse-Fine Convolutional Deep-Learning Strategy for Human Activity Recognition. Sensors, 19.","DOI":"10.3390\/s19071556"},{"key":"ref_18","unstructured":"Chen, C., Kehtarnavaz, N., and Jafari, R. (2014, January 26\u201330). A medication adherence monitoring system for pill bottles based on a wearable inertial sensor. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"103","DOI":"10.3233\/AIS-2009-0016","article-title":"Distributed recognition of human actions using wearable motion sensor networks","volume":"1","author":"Yang","year":"2009","journal-title":"J. Ambient Intell. Smart Environ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1109\/MSP.2018.2846286","article-title":"A Survey on Smart Homes for Aging in Place: Toward Solutions to the Specific Needs of the Elderly","volume":"35","author":"Nathan","year":"2018","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1427","DOI":"10.1109\/JIOT.2018.2856119","article-title":"Orientation independent activity\/gesture recognition using wearable motion sensors","volume":"6","author":"Wu","year":"2018","journal-title":"IEEE Internet Things J."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Liu, J., Wang, Z., Zhong, L., Wickramasuriya, J., and Vasudevan, V. (2009, January 9\u201313). uWave: Accelerometer-Based Personalized Gesture Recognition and Its Applications. Proceedings of the Seventh Annual IEEE International Conference on Pervasive Computing and Communications (PerCom 2009), Galveston, TX, USA.","DOI":"10.1109\/PERCOM.2009.4912759"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Alves, J., Silva, J., Grifo, E., Resende, C., and Sousa, I. (2019). Wearable Embedded Intelligence for Detection of Falls Independently of on-Body Location. Sensors, 19.","DOI":"10.3390\/s19112426"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/THMS.2014.2362520","article-title":"Improving human action recognition using fusion of depth camera and inertial sensors","volume":"45","author":"Chen","year":"2015","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1109\/JSEN.2015.2487358","article-title":"A real-time human action recognition system using depth and inertial sensor fusion","volume":"16","author":"Chen","year":"2016","journal-title":"IEEE Sens. J."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4405","DOI":"10.1007\/s11042-015-3177-1","article-title":"A survey of depth and inertial sensor fusion for human action recognition","volume":"76","author":"Chen","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Dawar, N., and Kehtarnavaz, N. (2018, January 12\u201315). A convolutional neural network-based sensor fusion system for monitoring transition movements in healthcare applications. Proceedings of the IEEE 14th International Conference on Control and Automation, Anchorage, AK, USA.","DOI":"10.1109\/ICCA.2018.8444326"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"9660","DOI":"10.1109\/JSEN.2018.2872862","article-title":"Action detection and recognition in continuous action streams by deep learning-based sensing fusion","volume":"18","author":"Dawar","year":"2018","journal-title":"IEEE Sens. J."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Rwigema, J., Choi, H.R., and Kim, T. (2019). A Differential Evolution Approach to Optimize Weights of Dynamic Time Warping for Multi-Sensor Based Gesture Recognition. Sensors, 19.","DOI":"10.3390\/s19051007"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"7019","DOI":"10.1109\/ACCESS.2017.2788558","article-title":"Real-time continuous detection and recognition of subject-specific smart tv gestures via fusion of depth and inertial sensing","volume":"6","author":"Dawar","year":"2018","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Chen, C., Jafari, R., and Kehtarnavaz, N. (2015, January 27\u201330). UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. Proceedings of the 2015 IEEE International Conference on Image Processing, Quebec City, QC, Canada.","DOI":"10.1109\/ICIP.2015.7350781"},{"key":"ref_32","unstructured":"Chen, C., Liu, K., Jafari, R., and Kehtarnavaz, N. (2014, January 26\u201330). Home-based senior fitness test measurement system using collaborative inertial and depth sensors. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wei, H., and Kehtarnavaz, N. (2018, January 21\u201323). Determining Number of Speakers from Single Microphone Speech Signals by Multi-Label Convolutional Neural Network. Proceedings of the 44th Annual Conference of the IEEE Industrial Electronics Society, Washington, DC, USA.","DOI":"10.1109\/IECON.2018.8592773"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Tao, F., and Busso, C. (2018, January 23\u201327). Aligning audiovisual features for audiovisual speech recognition. Proceedings of the IEEE International Conference on Multimedia and Expo, San Diego, CA, USA.","DOI":"10.1109\/ICME.2018.8486455"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, Z., Kong, Z., Changra, S., Tao, H., and Khan, L. (2019, January 8\u201311). Robust High Dimensional Stream Classification with Novel Class Detection. Proceedings of the IEEE 35th International Conference on Data Engineering, Macao, China.","DOI":"10.1109\/ICDE.2019.00128"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/17\/3680\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:13:43Z","timestamp":1760188423000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/17\/3680"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,24]]},"references-count":36,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2019,9]]}},"alternative-id":["s19173680"],"URL":"https:\/\/doi.org\/10.3390\/s19173680","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,8,24]]}}}