{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T11:54:13Z","timestamp":1791806053584,"version":"4.3.4"},"reference-count":39,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2022,10,21]],"date-time":"2022-10-21T00:00:00Z","timestamp":1666310400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human gait analysis presents an opportunity to study complex spatiotemporal data transpiring as co-movement patterns of multiple moving objects (i.e., human joints). Such patterns are acknowledged as movement signatures specific to an individual, offering the possibility to identify each individual based on unique gait patterns. We present a spatiotemporal deep learning model, dubbed ST-DeepGait, to featurize spatiotemporal co-movement patterns of human joints, and accordingly classify such patterns to enable human gait recognition. To this end, the ST-DeepGait model architecture is designed according to the spatiotemporal human skeletal graph in order to impose learning the salient local spatial dynamics of gait as they occur over time. Moreover, we employ a multi-layer RNN architecture to induce a sequential notion of gait cycles in the model. Our experimental results show that ST-DeepGait can achieve recognition accuracy rates over 90%. Furthermore, we qualitatively evaluate the model with the class embeddings to show interpretable separability of the features in geometric latent space. Finally, to evaluate the generalizability of our proposed model, we perform a zero-shot detection on 10 classes of data completely unseen during training and achieve a recognition accuracy rate of 88% overall. With this paper, we also contribute our gait dataset captured with an RGB-D sensor containing approximately 30 video samples of each subject for 100 subjects totaling 3087 samples. While we use human gait analysis as a motivating application to evaluate ST-DeepGait, we believe that this model can be simply adopted and adapted to study co-movement patterns of multiple moving objects in other applications such as in sports analytics and traffic pattern analysis.<\/jats:p>","DOI":"10.3390\/s22208075","type":"journal-article","created":{"date-parts":[[2022,10,24]],"date-time":"2022-10-24T10:09:23Z","timestamp":1666606163000},"page":"8075","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["ST-DeepGait: A Spatiotemporal Deep Learning Model for Human Gait Recognition"],"prefix":"10.3390","volume":"22","author":[{"given":"Latisha","family":"Konz","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, University of Colorado Denver, Denver, CO 80204, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Hill","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Colorado Denver, Denver, CO 80204, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4102-9873","authenticated-orcid":false,"given":"Farnoush","family":"Banaei-Kashani","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Colorado Denver, Denver, CO 80204, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"201","DOI":"10.3758\/BF03212378","article-title":"Visual perception of biological motion and a model for its analysis","volume":"14","author":"Johansson","year":"1973","journal-title":"Percept. Psychophys."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1002\/(SICI)1099-0720(199912)13:6<513::AID-ACP616>3.0.CO;2-8","article-title":"Visual analysis of gait as a cue to identity","volume":"13","author":"Stevenage","year":"1999","journal-title":"Appl. Cogn. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Boyd, J.E., and Little, J.J. (2005). Biometric Gait Recognition, Springer.","DOI":"10.1007\/11493648_2"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Steinmetzer, T., Bonninger, I., Priwitzer, B., Reinhardt, F., Reckhardt, M.C., Erk, D., and Travieso, C.M. (2018, January 18\u201320). Clustering of Human Gait with Parkinson\u2019s Disease by Using Dynamic Time Warping. Proceedings of the 2018 IEEE International Work Conference on Bioinspired Intelligence (IWOBI), San Carlos, Alajuela, Costa Rica.","DOI":"10.1109\/IWOBI.2018.8464203"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Staranowicz, A., Brown, G.R., and Mariottini, G.L. (2013, January 29\u201331). Evaluating the Accuracy of a Mobile Kinect-based Gait-monitoring System for Fall Prediction. Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA \u201913), Rhodes, Greece.","DOI":"10.1145\/2504335.2504396"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Blumrosen, G., Miron, Y., Intrator, N., and Plotnik, M. (2016). A Real-Time Kinect Signature-Based Patient Home Monitoring System. Sensors, 16.","DOI":"10.3390\/s16111965"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1505","DOI":"10.1109\/TPAMI.2003.1251144","article-title":"Silhouette analysis-based gait recognition for human identification","volume":"25","author":"Wang","year":"2003","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","unstructured":"Collins, R., Gross, R., and Shi, J. (2002, January 20\u201321). Silhouette-based human identification from body shape and gait. Proceedings of the Fifth IEEE International Conference on Automatic Face Gesture Recognition, Washinton, DC, USA."},{"key":"ref_9","unstructured":"Yu, S., Tan, D., and Tan, T. (2006, January 20\u201324). A Framework for Evaluating the Effect of View Angle, Clothing and Carrying Condition on Gait Recognition. Proceedings of the 18th International Conference on Pattern Recognition (ICPR\u201906), Hong Kong, China."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1007\/s00371-015-1092-0","article-title":"DTW-based kernel and rank-level fusion for 3D gait recognition using Kinect","volume":"31","author":"Ahmed","year":"2015","journal-title":"Vis. Comput."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Andersson, V.O., and Araujo, R.M. (2015, January 25\u201330). Person identification using anthropometric and gait data from kinect sensor. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA.","DOI":"10.1609\/aaai.v29i1.9212"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"336","DOI":"10.2106\/00004623-198062030-00004","article-title":"The development of mature gait","volume":"62","author":"Sutherland","year":"1980","journal-title":"J. Bone Jt. Surg. Am. Vol."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Jain, L.C., and Medsker, L.R. (1999). Recurrent Neural Networks: Design and Applications, CRC Press, Inc.. [1st ed.].","DOI":"10.1201\/9781420049176"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Jain, A., Zamir, A.R., Savarese, S., and Saxena, A. (2015). Structural-RNN: Deep Learning on Spatio-Temporal Graphs. arXiv.","DOI":"10.1109\/CVPR.2016.573"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Fragkiadaki, K., Levine, S., Felsen, P., and Malik, J. (2015, January 7\u201313). Recurrent network models for human dynamics. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.494"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1109\/TPAMI.2005.39","article-title":"The humanID gait challenge problem: Data sets, performance, and analysis","volume":"27","author":"Sarkar","year":"2005","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1109\/TPAMI.2006.38","article-title":"Individual recognition using gait energy image","volume":"28","author":"Han","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chung, D., Tahboub, K., and Delp, E.J. (2017, January 22\u201329). A Two Stream Siamese Convolutional Neural Network for Person Re-identification. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.218"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Schroff, F., Kalenichenko, D., and Philbin, J. (2015). FaceNet: A Unified Embedding for Face Recognition and Clustering. arXiv.","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1007\/s10851-014-0501-8","article-title":"Dynamic distance-based shape features for gait recognition","volume":"50","author":"Whytock","year":"2014","journal-title":"J. Math. Imaging Vis."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pavllo, D., Feichtenhofer, C., Auli, M., and Grangier, D. (2019). Modeling Human Motion with Quaternion-based Neural Networks. arXiv.","DOI":"10.1007\/s11263-019-01245-6"},{"key":"ref_22","unstructured":"Horst, F., Lapuschkin, S., Samek, W., M\u00fcller, K., and Sch\u00f6llhorn, W.I. (2018). What is Unique in Individual Gait Patterns? Understanding and Interpreting Deep Learning in Gait Analysis. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1016\/j.patrec.2015.06.020","article-title":"A framework for gait-based recognition using Kinect","volume":"68","author":"Kastaniotis","year":"2015","journal-title":"Pattern Recognit. Lett."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Dikovski, B., Madjarov, G., and Gjorgjevikj, D. (2014, January 26\u201330). Evaluation of different feature sets for gait recognition using skeletal data from Kinect. Proceedings of the 2014 37th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia.","DOI":"10.1109\/MIPRO.2014.6859769"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jiang, S., Wang, Y., Zhang, Y., and Sun, J. (2014, January 1\u20135). Real time gait recognition system based on kinect skeleton feature. Proceedings of the Asian Conference on Computer Vision, Singapore.","DOI":"10.1007\/978-3-319-16628-5_4"},{"key":"ref_26","unstructured":"Mu, X., and Wu, Q. (2003, January 4\u20136). A complete dynamic model of five-link bipedal walking. Proceedings of the 2003 American Control Conference, Denver, CO, USA."},{"key":"ref_27","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, The MIT Press."},{"key":"ref_28","unstructured":"Simonyan, K., and Zisserman, A. (2014). Two-Stream Convolutional Networks for Action Recognition in Videos. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Holden, D., Saito, J., Komura, T., and Joyce, T. (2015, January 2\u20136). Learning motion manifolds with convolutional autoencoders. Proceedings of the SIGGRAPH Asia Technical Briefs, Kobe, Japan.","DOI":"10.1145\/2820903.2820918"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Dehzangi, O., Taherisadr, M., and ChangalVala, R. (2017). IMU-Based Gait Recognition Using Convolutional Neural Networks and Multi-Sensor Fusion. Sensors, 17.","DOI":"10.3390\/s17122735"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Martinez, J., Black, M.J., and Romero, J. (2017, January 21\u201326). On human motion prediction using recurrent neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.497"},{"key":"ref_32","unstructured":"Koppula, H.S., and Saxena, A. (2019, March 15). Learning Spatio-Temporal Structure from RGB-D Videos for Human Activity Detection and Anticipation. Available online: https:\/\/proceedings.mlr.press\/v28\/koppula13.html."},{"key":"ref_33","unstructured":"Karpatne, A., Watkins, W., Read, J.S., and Kumar, V. (2017). Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modeling. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_35","unstructured":"Olver, F.W.J., Daalhuis, A.B.O., Lozier, D.W., Schneider, B.I., Boisvert, R.F., Clark, C.W., Miller, B.R., and Saunders, B.V. (2019, March 15). NIST Digital Library of Mathematical Functions; Release 1.0.22, Available online: http:\/\/dlmf.nist.gov\/."},{"key":"ref_36","unstructured":"Konz, L., Hill, A., and Banaei-Kashani, F. (2019, March 15). CU Denver Gait Dataset. Available online: https:\/\/cse.ucdenver.edu\/~bdlab\/datasets\/gait\/."},{"key":"ref_37","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Masci, J., Meier, U., Cire\u00b8san, D.C., and Schmidhuber, J. (2019, March 15). Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction. Technical Report. Available online: https:\/\/people.idsia.ch\/~ciresan\/data\/icann2011.pdf.","DOI":"10.1007\/978-3-642-21735-7_7"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"J. Comput. Appl. Math."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/20\/8075\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:59:14Z","timestamp":1760144354000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/20\/8075"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,21]]},"references-count":39,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["s22208075"],"URL":"https:\/\/doi.org\/10.3390\/s22208075","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,21]]}}}