{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T04:36:52Z","timestamp":1778647012197,"version":"3.51.4"},"reference-count":40,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2024,3,16]],"date-time":"2024-03-16T00:00:00Z","timestamp":1710547200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Science and ICT (MSIT)","award":["IITP-2023-RS-2023-00156354"],"award-info":[{"award-number":["IITP-2023-RS-2023-00156354"]}]},{"name":"Ministry of Science and ICT (MSIT)","award":["IITP-2023-RS-2023-00254529"],"award-info":[{"award-number":["IITP-2023-RS-2023-00254529"]}]},{"name":"Institute for Information and Communications Technology Planning and Evaluation (IITP)","award":["IITP-2023-RS-2023-00156354"],"award-info":[{"award-number":["IITP-2023-RS-2023-00156354"]}]},{"name":"Institute for Information and Communications Technology Planning and Evaluation (IITP)","award":["IITP-2023-RS-2023-00254529"],"award-info":[{"award-number":["IITP-2023-RS-2023-00254529"]}]},{"name":"Korean government (MSIT)","award":["IITP-2023-RS-2023-00156354"],"award-info":[{"award-number":["IITP-2023-RS-2023-00156354"]}]},{"name":"Korean government (MSIT)","award":["IITP-2023-RS-2023-00254529"],"award-info":[{"award-number":["IITP-2023-RS-2023-00254529"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Physical exercise affects many facets of life, including mental health, social interaction, physical fitness, and illness prevention, among many others. Therefore, several AI-driven techniques have been developed in the literature to recognize human physical activities. However, these techniques fail to adequately learn the temporal and spatial features of the data patterns. Additionally, these techniques are unable to fully comprehend complex activity patterns over different periods, emphasizing the need for enhanced architectures to further increase accuracy by learning spatiotemporal dependencies in the data individually. Therefore, in this work, we develop an attention-enhanced dual-stream network (PAR-Net) for physical activity recognition with the ability to extract both spatial and temporal features simultaneously. The PAR-Net integrates convolutional neural networks (CNNs) and echo state networks (ESNs), followed by a self-attention mechanism for optimal feature selection. The dual-stream feature extraction mechanism enables the PAR-Net to learn spatiotemporal dependencies from actual data. Furthermore, the incorporation of a self-attention mechanism makes a substantial contribution by facilitating targeted attention on significant features, hence enhancing the identification of nuanced activity patterns. The PAR-Net was evaluated on two benchmark physical activity recognition datasets and achieved higher performance by surpassing the baselines comparatively. Additionally, a thorough ablation study was conducted to determine the best optimal model for human physical activity recognition.<\/jats:p>","DOI":"10.3390\/s24061908","type":"journal-article","created":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T04:25:15Z","timestamp":1710735915000},"page":"1908","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["PAR-Net: An Enhanced Dual-Stream CNN\u2013ESN Architecture for Human Physical Activity Recognition"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0565-2935","authenticated-orcid":false,"given":"Imran Ullah","family":"Khan","sequence":"first","affiliation":[{"name":"Mixed Reality and Interaction Lab, Department of Software, Sejong University, Seoul 05006, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9425-2601","authenticated-orcid":false,"given":"Jong Weon","family":"Lee","sequence":"additional","affiliation":[{"name":"Mixed Reality and Interaction Lab, Department of Software, Sejong University, Seoul 05006, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,3,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Taha, A., Zayed, H.H., Khalifa, M., and El-Horbaty, E.-S.M. (2015, January 12\u201315). Human activity recognition for surveillance applications. Proceedings of the 7th International Conference on Information Technology, Amman, Jordan.","DOI":"10.15849\/icit.2015.0103"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1529","DOI":"10.1007\/s00371-016-1296-y","article-title":"Percentage of human-occupied areas for fall detection from two views","volume":"33","author":"Mousse","year":"2017","journal-title":"Vis. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.eswa.2019.04.057","article-title":"A survey on wearable sensor modality centred human activity recognition in health care","volume":"137","author":"Wang","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.1109\/JBHI.2018.2840834","article-title":"Depth-camera-based system for estimating energy expenditure of physical activities in gyms","volume":"23","author":"Lin","year":"2018","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Obdr\u017e\u00e1lek, \u0160., Kurillo, G., Ofli, F., Bajcsy, R., Seto, E., Jimison, H., and Pavel, M. (September, January 28). Accuracy and robustness of Kinect pose estimation in the context of coaching of elderly population. Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA, USA.","DOI":"10.1109\/EMBC.2012.6346149"},{"key":"ref_6","unstructured":"Aoki, H., Miyazaki, M., Nakamura, H., Furukawa, R., Sagawa, R., and Kawasaki, H. (2012, January 20\u201323). Non-contact respiration measurement using structured light 3-D sensor. Proceedings of the 2012 Proceedings of SICE Annual Conference (SICE), Akita, Japan."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"11716","DOI":"10.1109\/ACCESS.2021.3051375","article-title":"Action recognition from thermal videos using joint and skeleton information","volume":"9","author":"Batchuluun","year":"2021","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Bulbul, M.F., Tabussum, S., Ali, H., Zheng, W., Lee, M.Y., and Ullah, A. (2021). Exploring 3D human action recognition using STACOG on multi-view depth motion maps sequences. Sensors, 21.","DOI":"10.3390\/s21113642"},{"key":"ref_9","first-page":"3200","article-title":"Human action recognition from various data modalities: A review","volume":"45","author":"Sun","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"106970","DOI":"10.1016\/j.knosys.2021.106970","article-title":"A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions","volume":"223","author":"Yadav","year":"2021","journal-title":"Knowl.-Based Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"107102","DOI":"10.1016\/j.asoc.2021.107102","article-title":"Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications","volume":"103","author":"Ullah","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1109\/THMS.2021.3138708","article-title":"A multi-stream sequence learning framework for human interaction recognition","volume":"52","author":"Haroon","year":"2022","journal-title":"IEEE Trans. Human-Mach. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Pham, Q.-T., Nguyen, D.-A., Nguyen, T.-T., Nguyen, T.N., Nguyen, D.-T., Pham, D.-T., Tran, T.H., Le, T.-L., and Vu, H. (2022, January 1\u20133). A study on skeleton-based action recognition and its application to physical exercise recognition. Proceedings of the 11th International Symposium on Information and Communication Technology, Hanoi, Vietnam.","DOI":"10.1145\/3568562.3568639"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sardari, S., Sharifzadeh, S., Daneshkhah, A., Nakisa, B., Loke, S.W., Palade, V., and Duncan, M.J. (2023). Artificial Intelligence for skeleton-based physical rehabilitation action evaluation: A systematic review. Comput. Biol. Med., 158.","DOI":"10.1016\/j.compbiomed.2023.106835"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"109231","DOI":"10.1016\/j.patcog.2022.109231","article-title":"SpatioTemporal focus for skeleton-based action recognition","volume":"136","author":"Wu","year":"2023","journal-title":"Pattern Recognit."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.patcog.2017.10.033","article-title":"Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition","volume":"76","author":"Nunez","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1007\/s12652-019-01301-6","article-title":"Skeleton-based comparison of throwing motion for handball players","volume":"11","author":"Elaoud","year":"2020","journal-title":"J. Ambient Intell. Humaniz. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"14604582231214589","DOI":"10.1177\/14604582231214589","article-title":"A comparative study of the 2D-and 3D-based skeleton avatar technology for assessing physical activity and functioning among healthy older adults","volume":"29","author":"Lincke","year":"2023","journal-title":"Health Inform. J."},{"key":"ref_19","unstructured":"Mart\u0131nez, G.H. (2019). Openpose: Whole-Body Pose Estimation. [Ph.D. Thesis, Carnegie Mellon University]."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Du, Y., Fu, Y., and Wang, L. (2015, January 3\u20136). Skeleton based action recognition with convolutional neural network. Proceedings of the 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ACPR.2015.7486569"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1586","DOI":"10.1109\/TIP.2017.2785279","article-title":"Skeleton-based human action recognition with global context-aware attention LSTM networks","volume":"27","author":"Liu","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.patcog.2017.02.030","article-title":"Enhanced skeleton visualization for view invariant human action recognition","volume":"68","author":"Liu","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1016\/j.patrec.2018.04.035","article-title":"Combining CNN streams of RGB-D and skeletal data for human activity recognition","volume":"115","author":"Khaire","year":"2018","journal-title":"Pattern Recognit. Lett."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2572","DOI":"10.1049\/iet-ipr.2019.0030","article-title":"Human activity recognition using 2D skeleton data and supervised machine learning","volume":"13","author":"Ghazal","year":"2019","journal-title":"IET Image Process."},{"key":"ref_25","unstructured":"Li, C., Wang, P., Wang, S., Hou, Y., and Li, W. (2017, January 10\u201314). Skeleton-based action recognition using LSTM and CNN. Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Hong Kong, China."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., and Tian, Q. (2019, January 15\u201320). Actional-structural graph convolutional networks for skeleton-based action recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00371"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1109\/TCE.2019.2908986","article-title":"Deep learning for recognizing human activities using motions of skeletal joints","volume":"65","author":"Phyo","year":"2019","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Nadeem, A., Jalal, A., and Kim, K. (2020). Accurate physical activity recognition using multidimensional features and Markov model for smart health fitness. Symmetry, 12.","DOI":"10.3390\/sym12111766"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tasnim, N., Islam, M.K., and Baek, J.-H. (2021). Deep learning based human activity recognition using spatio-temporal image formation of skeleton joints. Appl. Sci., 11.","DOI":"10.3390\/app11062675"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3458281","article-title":"Dual-stream structured graph convolution network for skeleton-based action recognition","volume":"17","author":"Xu","year":"2021","journal-title":"ACM Trans. Multimedia Comput. Commun. Appl."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Guerra, B.M.V., Ramat, S., Beltrami, G., and Schmid, M. (2023). Recurrent Network Solutions for Human Posture Recognition Based on Kinect Skeletal Data. Sensors, 23.","DOI":"10.3390\/s23115260"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"23019","DOI":"10.1109\/ACCESS.2023.3247583","article-title":"Periodic physical activity information segmentation, counting and recognition from video","volume":"11","author":"Cheng","year":"2023","journal-title":"IEEE Access"},{"key":"ref_33","first-page":"65","article-title":"Mono Camera-based Human Skeletal Tracking for Squat Exercise Abnormality Detection using Double Exponential Smoothing","volume":"13","author":"Muhammad","year":"2022","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"107135","DOI":"10.1109\/ACCESS.2023.3316009","article-title":"AI Trainer: Autoencoder Based Approach for Squat Analysis and Correction","volume":"11","author":"Chariar","year":"2023","journal-title":"IEEE Access"},{"key":"ref_35","first-page":"6650632","article-title":"3D skeletal human action recognition using a CNN fusion model","volume":"2021","author":"Li","year":"2021","journal-title":"Math. Probl. Eng."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Khan, I.U., Afzal, S., and Lee, J.W. (2022). Human activity recognition via hybrid deep learning based model. Sensors, 22.","DOI":"10.3390\/s22010323"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Luwe, Y.J., Lee, C.P., and Lim, K.M. (2022). Wearable sensor-based human activity recognition with hybrid deep learning model. Informatics, 9.","DOI":"10.3390\/informatics9030056"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"107398","DOI":"10.1016\/j.ymssp.2020.107398","article-title":"1D convolutional neural networks and applications: A survey","volume":"151","author":"Kiranyaz","year":"2021","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"109920","DOI":"10.1016\/j.petrol.2021.109920","article-title":"An echo state network with attention mechanism for production prediction in reservoirs","volume":"209","author":"Liu","year":"2022","journal-title":"J. Pet. Sci. Eng."},{"key":"ref_40","unstructured":"Tuameh, M. (2023, October 05). Physical Exercise Recognition dataset. Available online: https:\/\/www.kaggle.com\/datasets\/muhannadtuameh\/exercise-recognition-time-series."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/6\/1908\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:14:44Z","timestamp":1760105684000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/6\/1908"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,16]]},"references-count":40,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["s24061908"],"URL":"https:\/\/doi.org\/10.3390\/s24061908","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,16]]}}}