{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T16:42:16Z","timestamp":1783701736288,"version":"3.55.0"},"reference-count":52,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In recent years, Human Activity Recognition (HAR) has become one of the most important research topics in the domains of health and human-machine interaction. Many Artificial intelligence-based models are developed for activity recognition; however, these algorithms fail to extract spatial and temporal features due to which they show poor performance on real-world long-term HAR. Furthermore, in literature, a limited number of datasets are publicly available for physical activities recognition that contains less number of activities. Considering these limitations, we develop a hybrid model by incorporating Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) for activity recognition where CNN is used for spatial features extraction and LSTM network is utilized for learning temporal information. Additionally, a new challenging dataset is generated that is collected from 20 participants using the Kinect V2 sensor and contains 12 different classes of human physical activities. An extensive ablation study is performed over different traditional machine learning and deep learning models to obtain the optimum solution for HAR. The accuracy of 90.89% is achieved via the CNN-LSTM technique, which shows that the proposed model is suitable for HAR applications.<\/jats:p>","DOI":"10.3390\/s22010323","type":"journal-article","created":{"date-parts":[[2022,1,9]],"date-time":"2022-01-09T23:08:26Z","timestamp":1641769706000},"page":"323","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":202,"title":["Human Activity Recognition via Hybrid Deep Learning Based Model"],"prefix":"10.3390","volume":"22","author":[{"given":"Imran Ullah","family":"Khan","sequence":"first","affiliation":[{"name":"Mixed Reality and Interaction Lab, Department of Software, Sejong University, Seoul 05006, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sitara","family":"Afzal","sequence":"additional","affiliation":[{"name":"Mixed Reality and Interaction Lab, Department of Software, Sejong University, Seoul 05006, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9425-2601","authenticated-orcid":false,"given":"Jong Weon","family":"Lee","sequence":"additional","affiliation":[{"name":"Mixed Reality and Interaction Lab, Department of Software, Sejong University, Seoul 05006, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.1109\/JBHI.2018.2840834","article-title":"Depth-camera-based system for estimating energy expenditure of physical activities in gyms","volume":"23","author":"Lin","year":"2018","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_2","first-page":"577","article-title":"Human activity recognition for surveillance applications","volume":"1","author":"Taha","year":"2016","journal-title":"Hindawi"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1529","DOI":"10.1007\/s00371-016-1296-y","article-title":"Percentage of human-occupied areas for fall detection from two views","volume":"33","author":"Mousse","year":"2017","journal-title":"Vis. Comput."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"107102","DOI":"10.1016\/j.asoc.2021.107102","article-title":"Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications","volume":"103","author":"Ullah","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1016\/j.neucom.2019.12.151","article-title":"Conflux LSTMs network: A novel approach for multi-view action recognition","volume":"435","author":"Ullah","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1123\/japa.16.4.369","article-title":"Physical activity measurement in older adults: Relationships with mental health","volume":"16","author":"Parker","year":"2008","journal-title":"J. Aging Phys. Act."},{"key":"ref_7","unstructured":"WHO (2021, November 05). World Health Organization. Available online: https:\/\/www.who.int\/emergencies\/diseases\/novel-coronavirus-2019."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2572","DOI":"10.1049\/iet-ipr.2019.0030","article-title":"Human activity recognition using 2D skeleton data and supervised machine learning","volume":"13","author":"Ghazal","year":"2019","journal-title":"IET Image Processing"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhu, G., Zhang, L., Shen, P., and Song, J. (2016). An online continuous human action recognition algorithm based on the Kinect sensor. Sensors, 16.","DOI":"10.3390\/s16020161"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Manzi, A., Dario, P., and Cavallo, F. (2017). A human activity recognition system based on dynamic clustering of skeleton data. Sensors, 17.","DOI":"10.3390\/s17051100"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1049\/iet-cvi.2017.0062","article-title":"Skeleton-based human activity recognition for elderly monitoring systems","volume":"12","author":"Hbali","year":"2018","journal-title":"IET Comput. Vis."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zanfir, M., Leordeanu, M., and Sminchisescu, C. (2013, January 3\u20136). The Moving Pose: An Efficient 3d Kinematics Descriptor for Low-Latency Action Recognition and Detection. Proceedings of the Computer Vision Foundation, ICCV, Computer Vison Foundation, Sydney, Australia.","DOI":"10.1109\/ICCV.2013.342"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.jvcir.2013.04.007","article-title":"Sequence of the most informative joints (smij): A new representation for human skeletal action recognition","volume":"25","author":"Ofli","year":"2014","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yang, X., and Tian, Y.L. (2021, January 16\u201321). Eigenjoints-Based Action Recognition Using Naive-Bayes-Nearest-Neighbor. Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA.","DOI":"10.1109\/CVPRW.2012.6239232"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1750008","DOI":"10.1142\/S0218001417500082","article-title":"Automatic learning of articulated skeletons based on mean of 3D joints for efficient action recognition","volume":"31","author":"Tamou","year":"2017","journal-title":"Int. J. Pattern Recognit. Artif. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/TMM.2015.2505089","article-title":"Effective active skeleton representation for low latency human action recognition","volume":"18","author":"Cai","year":"2015","journal-title":"IEEE Trans. Multimed."},{"key":"ref_17","unstructured":"Hussein, M.E., Torki, M., Gowayyed, M.A., and El-Saban, M. (2014, January 23\u201328). Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3d Joint Locations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA."},{"key":"ref_18","first-page":"101","article-title":"An Efficient Data Augmentation CNN-Network for Skeleton-based Human Action Recognition","volume":"7","author":"Arthi","year":"2020","journal-title":"IJARTET J."},{"key":"ref_19","first-page":"101964","article-title":"On human motion prediction using recurrent neural networks","volume":"86","author":"Martinez","year":"2021","journal-title":"Comput. Aided Geom. Des."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, C., Zhong, Q., Xie, D., and Pu, S. (2018, January 18\u201323). Co-Occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation. Proceedings of the Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.24963\/ijcai.2018\/109"},{"key":"ref_21","first-page":"10010","article-title":"Symbiotic graph neural networks for 3d skeleton-based human action recognition and motion prediction","volume":"1","author":"Li","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1080\/00031305.1992.10475879","article-title":"An introduction to kernel and nearest-neighbor nonparametric regression","volume":"46","author":"Altman","year":"1992","journal-title":"Am. Stat."},{"key":"ref_24","first-page":"65","article-title":"Matching and prediction on the principle of biological classification","volume":"8","author":"Belson","year":"1959","journal-title":"J. R. Stat. Soc. Ser. C (Appl. Stat.)"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1111\/j.1469-1809.1936.tb02137.x","article-title":"The use of multiple measurements in taxonomic problems","volume":"7","author":"Fisher","year":"1936","journal-title":"Annu. Eugen."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Webb, G.I., Sammut, C., and Perlich, C. (2011). Lazy Learning. Encyclopedia of Machine Learning, Springer Science & Business Media. [1st ed.].","DOI":"10.1007\/978-0-387-30164-8_443"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1613\/jair.614","article-title":"Popular ensemble methods: An empirical study","volume":"11","author":"Opitz","year":"1999","journal-title":"J. Artif. Intell. Res."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2554","DOI":"10.1073\/pnas.79.8.2554","article-title":"Neural Networks and Physical Systems with Emergent Collective Computational Abilities","volume":"79","author":"Ilopfield","year":"1982","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hur, T., Bang, J., Lee, J., Kim, J.-I., and Lee, S.J.S. (2018). Iss2Image: A novel signal-encoding technique for CNN-based human activity recognition. Sensors, 18.","DOI":"10.3390\/s18113910"},{"key":"ref_30","first-page":"1137","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"39","author":"Ren","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"114177","DOI":"10.1016\/j.eswa.2020.114177","article-title":"MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach","volume":"167","author":"Kwon","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_32","first-page":"4039","article-title":"1D-CNN: Speech Emotion Recognition System Using a Stacked Network with Dilated CNN Features","volume":"67","author":"Mustaqeem","year":"2021","journal-title":"CMC-Comput. Mater. Contin."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"8968","DOI":"10.1109\/TIP.2021.3116790","article-title":"Light-DehazeNet: A Novel Lightweight CNN Architecture for Single Image Dehazing","volume":"30","author":"Ullah","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1729","DOI":"10.1007\/s11554-020-01020-8","article-title":"SD-Net: Understanding overcrowded scenes in real-time via an efficient dilated convolutional neural network","volume":"18","author":"Khan","year":"2021","journal-title":"J. Real-Time Image Process."},{"key":"ref_35","unstructured":"Li, X., Zhang, Y., Zhang, J., Chen, S., Marsic, I., Farneth, R.A., and Burd, R.S. (2021, November 30). Concurrent Activity Recognition with Multimodal CNN-LSTM Structure. Available online: https:\/\/arxiv.org\/ftp\/arxiv\/papers\/1702\/1702.01638.pdf."},{"key":"ref_36","first-page":"100046","article-title":"Deep learning based human activity recognition (HAR) using wearable sensor data","volume":"1","author":"Gupta","year":"2021","journal-title":"Int. J. Inf. Manag. Data Insights"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ullah, W., Ullah, A., Hussain, T., Khan, Z.A., and Baik, S.W. (2021). An Efficient Anomaly Recognition Framework Using an Attention Residual LSTM in Surveillance Videos. Sensors, 21.","DOI":"10.3390\/s21082811"},{"key":"ref_38","first-page":"1001","article-title":"Artificial Intelligence of Things-assisted two-stream neural network for anomaly detection in surveillance Big Video Data","volume":"1","author":"Ullah","year":"2021","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2133","DOI":"10.3390\/math8122133","article-title":"CLSTM: Deep feature-based speech emotion recognition using the hierarchical ConvLSTM network","volume":"8","author":"Kwon","year":"2020","journal-title":"Mathematics"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"820","DOI":"10.1016\/j.future.2021.06.045","article-title":"Human action recognition using attention based LSTM network with dilated CNN features","volume":"125","author":"Muhammad","year":"2021","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_41","first-page":"1","article-title":"Deep-ReID: Deep features and autoencoder assisted image patching strategy for person re-identification in smart cities surveillance","volume":"1","author":"Khan","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Khan, S.U., Haq, I.U., Khan, Z.A., Khan, N., Lee, M.Y., and Baik, S.W. (2021). Atrous Convolutions and Residual GRU Based Architecture for Matching Power Demand with Supply. Sensors, 21.","DOI":"10.3390\/s21217191"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"7797","DOI":"10.1109\/ACCESS.2020.3047732","article-title":"Batteries state of health estimation via efficient neural networks with multiple channel charging profiles","volume":"9","author":"Khan","year":"2020","journal-title":"IEEE Access"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Sajjad, M., Khan, S.U., Khan, N., Haq, I.U., Ullah, A., Lee, M.Y., and Baik, S.W. (2020). Towards efficient building designing: Heating and cooling load prediction via multi-output model. Sensors, 20.","DOI":"10.3390\/s20226419"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Khan, Z.A., Hussain, T., Ullah, A., Rho, S., Lee, M., and Baik, S.W. (2020). Towards efficient electricity forecasting in residential and commercial buildings: A novel hybrid CNN with a LSTM-AE based framework. Sensors, 20.","DOI":"10.3390\/s20051399"},{"key":"ref_46","first-page":"1","article-title":"Beyond english-centric multilingual machine translation","volume":"22","author":"Fan","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Mekruksavanich, S., and Jitpattanakul, A.J.E. (2021). Deep convolutional neural network with rnns for complex activity recognition using wrist-worn wearable sensor data. Electronics, 10.","DOI":"10.3390\/electronics10141685"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Khan, N., Ullah, F.U.M., Haq, I.U., Khan, S.U., Lee, M.Y., and Baik, S.W. (2021). AB-Net: A Novel Deep Learning Assisted Framework for Renewable Energy Generation Forecasting. Mathematics, 9.","DOI":"10.3390\/math9192456"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"107023","DOI":"10.1016\/j.ijepes.2021.107023","article-title":"DB-Net: A novel dilated CNN based multi-step forecasting model for power consumption in integrated local energy systems","volume":"133","author":"Khan","year":"2021","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Khan, S.U., and Baik, R. (2020). MPPIF-net: Identification of plasmodium falciparum parasite mitochondrial proteins using deep features with multilayer Bi-directional LSTM. Processes, 8.","DOI":"10.3390\/pr8060725"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Haq, I.U., Ullah, A., Khan, S.U., Khan, N., Lee, M.Y., Rho, S., and Baik, S.W. (2021). Sequential learning-based energy consumption prediction model for residential and commercial sectors. Mathematics, 9.","DOI":"10.3390\/math9060605"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Ullah, F.U.M., Khan, N., Hussain, T., Lee, M.Y., and Baik, S.W. (2021). Diving Deep into Short-Term Electricity Load Forecasting: Comparative Analysis and a Novel Framework. Mathematics, 9.","DOI":"10.3390\/math9060611"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/323\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:12:29Z","timestamp":1760364749000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/323"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,1]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22010323"],"URL":"https:\/\/doi.org\/10.3390\/s22010323","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,1]]}}}