{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:06:38Z","timestamp":1787029598145,"version":"build-2736575974"},"reference-count":43,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,1,5]],"date-time":"2022-01-05T00:00:00Z","timestamp":1641340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Sciences and Engineering Research 493 Council (NSERC) Discovery Grant funding","award":["10007544"],"award-info":[{"award-number":["10007544"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Motion capture sensor-based gait emotion recognition is an emerging sub-domain of human emotion recognition. Its applications span a variety of fields including smart home design, border security, robotics, virtual reality, and gaming. In recent years, several deep learning-based approaches have been successful in solving the Gait Emotion Recognition (GER) problem. However, a vast majority of such methods rely on Deep Neural Networks (DNNs) with a significant number of model parameters, which lead to model overfitting as well as increased inference time. This paper contributes to the domain of knowledge by proposing a new lightweight bi-modular architecture with handcrafted features that is trained using a RMSprop optimizer and stratified data shuffling. The method is highly effective in correctly inferring human emotions from gait, achieving a micro-mean average precision of 0.97 on the Edinburgh Locomotive Mocap Dataset. It outperforms all recent deep-learning methods, while having the lowest inference time of 16.3 milliseconds per gait sample. This research study is beneficial to applications spanning various fields, such as emotionally aware assistive robotics, adaptive therapy and rehabilitation, and surveillance.<\/jats:p>","DOI":"10.3390\/s22010403","type":"journal-article","created":{"date-parts":[[2022,1,9]],"date-time":"2022-01-09T23:08:26Z","timestamp":1641769706000},"page":"403","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Motion Capture Sensor-Based Emotion Recognition Using a Bi-Modular Sequential Neural Network"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1809-3108","authenticated-orcid":false,"given":"Yajurv","family":"Bhatia","sequence":"first","affiliation":[{"name":"Department of Computer Science, Faculty of Science, Main Campus, University of Calgary, Calgary, AB T2N 1N4, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1850-4816","authenticated-orcid":false,"given":"ASM Hossain","family":"Bari","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Science, Main Campus, University of Calgary, Calgary, AB T2N 1N4, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gee-Sern Jison","family":"Hsu","sequence":"additional","affiliation":[{"name":"Artificial Vision Laboratory, Department of Mechanical Engineering, National Taiwan University of Science and Technology, No. 1, Sec. 4, Roosevelt Rd., Taipei 10617, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marina","family":"Gavrilova","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Science, Main Campus, University of Calgary, Calgary, AB T2N 1N4, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Gavrilova, M.L., Ahmed, F., Bari, A.H., Liu, R., Liu, T., Maret, Y., Sieu, B.K., and Sudhakar, T. (2021). Multi-modal motion-capture-based biometric systems for emergency response and patient rehabilitation. Research Anthology on Rehabilitation Practices and Therapy, IGI Global.","DOI":"10.4018\/978-1-7998-3432-8.ch032"},{"key":"ref_2","first-page":"1","article-title":"Cognitive intelligence: Deep learning, thinking, and reasoning by brain-inspired systems","volume":"10","author":"Wang","year":"2016","journal-title":"Int. J. Cogn. Inform. Nat. Intell."},{"key":"ref_3","unstructured":"Li, S., and Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput."},{"key":"ref_4","unstructured":"Xu, S., Fang, J., Hu, X., Ngai, E., Guo, Y., Leung, V., Cheng, J., and Hu, B. (2020). Emotion Recognition From Gait Analyses: Current Research and Future Directions. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ahmed, F., Sieu, B., and Gavrilova, M.L. (2018, January 16\u201318). Score and rank-level fusion for emotion recognition using genetic algorithm. Proceedings of the IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC), Berkeley, CA, USA.","DOI":"10.1109\/ICCI-CC.2018.8482086"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"11761","DOI":"10.1109\/ACCESS.2019.2963113","article-title":"Emotion recognition from body movement","volume":"8","author":"Ahmed","year":"2019","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Glowinski, D., Camurri, A., Volpe, G., Dael, N., and Scherer, K. (2008, January 23\u201328). Technique for automatic emotion recognition by body gesture analysis. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Anchorage, AK, USA.","DOI":"10.1109\/CVPRW.2008.4563173"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lugger, M., and Yang, B. (2007, January 15\u201320). The relevance of voice quality features in speaker independent emotion recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP\u201907, Honolulu, HI, USA.","DOI":"10.1109\/ICASSP.2007.367152"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1145\/1165255.1165259","article-title":"Emotion recognition from text using semantic labels and separable mixture models","volume":"5","author":"Wu","year":"2006","journal-title":"ACM Trans. Asian Lang. Inf. Process."},{"key":"ref_10","unstructured":"Castellano, G., Villalba, S.D., and Camurri, A. (2007, January 12\u201314). Recognising human emotions from body movement and gesture dynamics. Proceedings of the International Conference on Affective Computing and Intelligent Interaction, Lisbon, Portugal."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/BF00999605","article-title":"The identification of emotions from gait information","volume":"11","author":"Montepare","year":"1987","journal-title":"J. Nonverbal Behav."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1023\/B:JONB.0000023655.25550.be","article-title":"Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence","volume":"28","author":"Coulson","year":"2004","journal-title":"J. Nonverbal Behav."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"879","DOI":"10.1002\/(SICI)1099-0992(1998110)28:6<879::AID-EJSP901>3.0.CO;2-W","article-title":"Bodily expression of emotion","volume":"28","author":"Wallbott","year":"1998","journal-title":"Eur. J. Soc. Psychol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Destephe, M., Henning, A., Zecca, M., Hashimoto, K., and Takanishi, A. (2013, January 12\u201314). Perception of emotion and emotional intensity in humanoid robots gait. Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO), Shenzhen, China.","DOI":"10.1109\/ROBIO.2013.6739640"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhuang, J.R., Wu, G.Y., Lee, H.H., and Tanaka, E. (2019, January 4\u20138). Applying the Interaction of Walking-Emotion to an Assistive Device for Rehabilitation and Exercise. Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.","DOI":"10.1109\/IROS40897.2019.8968260"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Randhavane, T., Bera, A., Kapsaskis, K., Sheth, R., Gray, K., and Manocha, D. (2019, January 19\u201320). Eva: Generating emotional behavior of virtual agents using expressive features of gait and gaze. Proceedings of the ACM Symposium on Applied Perception, Barcelona, Spain.","DOI":"10.1145\/3343036.3343129"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1097\/00003086-199303000-00016","article-title":"Gait analysis. An essential tool in the treatment of cerebral palsy","volume":"288","author":"Gage","year":"1993","journal-title":"Clin. Orthop. Relat. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1136\/jnnp.2007.131045","article-title":"Parkinson\u2019s disease: Clinical features and diagnosis","volume":"79","author":"Jankovic","year":"2008","journal-title":"J. Neurol. Neurosurg. Psychiatry"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1108\/SR-10-2018-0270","article-title":"Motion capture sensing techniques used in human upper limb motion: A review","volume":"39","author":"Yahya","year":"2019","journal-title":"Sens. Rev."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Menolotto, M., Komaris, D.S., Tedesco, S., O\u2019Flynn, B., and Walsh, M. (2020). Motion Capture Technology in Industrial Applications: A Systematic Review. Sensors, 20.","DOI":"10.3390\/s20195687"},{"key":"ref_21","unstructured":"Artacho, B., and Savakis, A. (2021). OmniPose: A Multi-Scale Framework for Multi-Person Pose Estimation. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1109\/TPAMI.2019.2929257","article-title":"OpenPose: Realtime multi-person 2D pose estimation using Part Affinity Fields","volume":"43","author":"Cao","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Toshev, A., and Szegedy, C. (2014, January 23\u201328). Deeppose: Human pose estimation via deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.214"},{"key":"ref_24","first-page":"651","article-title":"Emotion models: A review","volume":"10","author":"Sreeja","year":"2017","journal-title":"Int. J. Control Theory Appl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/79.911197","article-title":"Emotion recognition in human-computer interaction","volume":"18","author":"Cowie","year":"2001","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1037\/0033-2909.99.2.143","article-title":"Vocal affect expression: A review and a model for future research","volume":"99","author":"Scherer","year":"1986","journal-title":"Psychol. Bull."},{"key":"ref_27","unstructured":"Davitz, J.R. (1964). The Communication of Emotional Meaning, Mcgraw Hill."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Ko, B.C. (2018). A Brief Review of Facial Emotion Recognition Based on Visual Information. Sensors, 18.","DOI":"10.3390\/s18020401"},{"key":"ref_29","unstructured":"Efron, D. (1941). Gesture and Environment, King\u2019s Crown Press."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kendon, A. (1983). The study of gesture: Some remarks on its history. Semiotics 1981, Springer.","DOI":"10.1007\/978-1-4615-9328-7_15"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"28","DOI":"10.3389\/frobt.2015.00028","article-title":"A review of human activity recognition methods","volume":"2","author":"Vrigkas","year":"2015","journal-title":"Front. Robot. AI"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1109\/TSMCB.2010.2044040","article-title":"Recognition of affect based on gait patterns","volume":"40","author":"Karg","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part B Cybern."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1109\/TAFFC.2016.2637343","article-title":"Identifying emotions from non-contact gaits information based on microsoft kinects","volume":"9","author":"Li","year":"2016","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e2364","DOI":"10.7717\/peerj.2364","article-title":"Emotion recognition using Kinect motion capture data of human gaits","volume":"4","author":"Li","year":"2016","journal-title":"PeerJ"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yan, S., Xiong, Y., and Lin, D. (2018, January 2\u20137). Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LO, USA.","DOI":"10.1609\/aaai.v32i1.12328"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Bhattacharya, U., Mittal, T., Chandra, R., Randhavane, T., Bera, A., and Manocha, D. (2020, January 7\u201312). Step: Spatial temporal graph convolutional networks for emotion perception from gaits. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i02.5490"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bhattacharya, U., Roncal, C., Mittal, T., Chandra, R., Kapsaskis, K., Gray, K., Bera, A., and Manocha, D. (2020, January 23\u201328). Take an emotion walk: Perceiving emotions from gaits using hierarchical attention pooling and affective mapping. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58607-2_9"},{"key":"ref_38","unstructured":"Randhavane, T., Bhattacharya, U., Kapsaskis, K., Gray, K., Bera, A., and Manocha, D. (2019). Identifying emotions from walking using affective and deep features. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Karg, M., Jenke, R., Seiberl, W., K\u00fchnlenz, K., Schwirtz, A., and Buss, M. (2009, January 10\u201312). A comparison of PCA, KPCA and LDA for feature extraction to recognize affect in gait kinematics. Proceedings of the 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, The Netherlands.","DOI":"10.1109\/ACII.2009.5349438"},{"key":"ref_40","unstructured":"Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., and Natsev, P. (2017). The Kinetics Human Action Video Dataset. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1007\/s00371-015-1092-0","article-title":"DTW-based kernel and rank-level fusion for 3D gait recognition using Kinect","volume":"31","author":"Ahmed","year":"2015","journal-title":"Vis. Comput."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"162708","DOI":"10.1109\/ACCESS.2019.2952065","article-title":"Artificial neural network based gait recognition using kinect sensor","volume":"7","author":"Bari","year":"2019","journal-title":"IEEE Access"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Habibie, I., Holden, D., Schwarz, J., Yearsley, J., and Komura, T. (2017, January 4\u20137). A recurrent variational autoencoder for human motion synthesis. Proceedings of the 28th British Machine Vision Conference, London, UK.","DOI":"10.5244\/C.31.119"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/403\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:36:26Z","timestamp":1760362586000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/403"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,5]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22010403"],"URL":"https:\/\/doi.org\/10.3390\/s22010403","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,5]]}}}