{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T13:28:26Z","timestamp":1778592506027,"version":"3.51.4"},"reference-count":59,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2020,3,16]],"date-time":"2020-03-16T00:00:00Z","timestamp":1584316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61701179"],"award-info":[{"award-number":["61701179"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["41604117"],"award-info":[{"award-number":["41604117"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004735","name":"Natural Science Foundation of\u00a0Hunan Province","doi-asserted-by":"publisher","award":["2019JJ50363"],"award-info":[{"award-number":["2019JJ50363"]}],"id":[{"id":"10.13039\/501100004735","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Hunan Provincial Key Lab of Intelligent Computing and Language Information Processing, China","award":["2018TP1018"],"award-info":[{"award-number":["2018TP1018"]}]},{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"publisher","award":["201808430285"],"award-info":[{"award-number":["201808430285"]}],"id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Gait recognition and understanding systems have shown a wide-ranging application prospect. However, their use of unstructured data from image and video has affected their performance, e.g., they are easily influenced by multi-views, occlusion, clothes, and object carrying conditions. This paper addresses these problems using a realistic 3-dimensional (3D) human structural data and sequential pattern learning framework with top-down attention modulating mechanism based on Hierarchical Temporal Memory (HTM). First, an accurate 2-dimensional (2D) to 3D human body pose and shape semantic parameters estimation method is proposed, which exploits the advantages of an instance-level body parsing model and a virtual dressing method. Second, by using gait semantic folding, the estimated body parameters are encoded using a sparse 2D matrix to construct the structural gait semantic image. In order to achieve time-based gait recognition, an HTM Network is constructed to obtain the sequence-level gait sparse distribution representations (SL-GSDRs). A top-down attention mechanism is introduced to deal with various conditions including multi-views by refining the SL-GSDRs, according to prior knowledge. The proposed gait learning model not only aids gait recognition tasks to overcome the difficulties in real application scenarios but also provides the structured gait semantic images for visual cognition. Experimental analyses on CMU MoBo, CASIA B, TUM-IITKGP, and KY4D datasets show a significant performance gain in terms of accuracy and robustness.<\/jats:p>","DOI":"10.3390\/s20061646","type":"journal-article","created":{"date-parts":[[2020,3,18]],"date-time":"2020-03-18T08:20:44Z","timestamp":1584519644000},"page":"1646","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Gait Recognition and Understanding Based on Hierarchical Temporal Memory Using 3D Gait Semantic Folding"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9899-874X","authenticated-orcid":false,"given":"Jian","family":"Luo","sequence":"first","affiliation":[{"name":"Hunan Provincial Key Laboratory of Intelligent Computing and Language Information Processing, Hunan Normal University, Changsha 410000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8612-735X","authenticated-orcid":false,"given":"Tardi","family":"Tjahjadi","sequence":"additional","affiliation":[{"name":"School of Engineering, University of Warwick, Gibbet Hill Road, Coventry CV4 7AL, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,3,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"70497","DOI":"10.1109\/ACCESS.2018.2879896","article-title":"Vision-Based Gait Recognition: A Survey","volume":"6","author":"Jasvinder","year":"2018","journal-title":"IEEE Access"},{"key":"ref_2","first-page":"102","article-title":"Multi-task GANs for View-Specific Feature Learning in Gait Recognition","volume":"14","author":"Yiwei","year":"2018","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1016\/j.patcog.2019.01.017","article-title":"A general tensor representation framework for cross-view gait recognition","volume":"90","author":"Ben","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Chao, H., He, Y., Zhang, J., and Feng, J. (February, January 27). Gaitset: Regarding gait as a set for cross-view gait recognition. Proceedings of the AAAI Conference on Artificial Intelligence 2019, Hawaii, HI, USA.","DOI":"10.1609\/aaai.v33i01.33018126"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1109\/TPAMI.2006.38","article-title":"Individual Recognition Using Gait Energy Image","volume":"28","author":"Han","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1186\/s41074-018-0039-6","article-title":"Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition","volume":"10","author":"Takemura","year":"2018","journal-title":"IPSJ Trans. Comput. Vis. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1016\/j.patcog.2017.02.014","article-title":"Fusion of spatial-temporal and kinematic features for gait recognition with deterministic learning","volume":"67","author":"Deng","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_8","unstructured":"Hawkins, J., Ahmad, S., and Dubinsky, D. (2020, March 15). Cortical Learning Algorithm and Hierarchical Temporal Memory. Numenta Whitepaper, 2011; pp. 1\u201368. Available online: http:\/\/numenta.org\/resources\/HTM_CorticalLearningAlgorithms.pdf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"111","DOI":"10.3389\/fncom.2017.00111","article-title":"The HTM Spatial Pooler\u2014A Neocortical Algorithm for Online Sparse Distributed Coding","volume":"11","author":"Yuwei","year":"2017","journal-title":"Front. Comput. Neurosci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1932","DOI":"10.1016\/j.neucom.2017.10.049","article-title":"Gait recognition via GEI subspace projections and collaborative representation classification","volume":"275","author":"Wei","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1016\/j.patrec.2009.04.012","article-title":"Frame difference energy image for gait recognition with incomplete silhouettes","volume":"30","author":"Chen","year":"2009","journal-title":"Pattern Recognit. Lett."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2052","DOI":"10.1016\/j.patrec.2010.05.027","article-title":"Gait recognition without subject cooperation","volume":"31","author":"Bashir","year":"2010","journal-title":"Pattern Recognit. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"973","DOI":"10.1016\/j.patcog.2010.10.011","article-title":"Gait flow image: A silhouette-based gait representation for human identification","volume":"44","author":"Lam","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"780","DOI":"10.1016\/j.sigpro.2011.09.022","article-title":"Gait recognition using Pose Kinematics and Pose Energy Image","volume":"92","author":"Roy","year":"2012","journal-title":"Signal Process."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1016\/j.patrec.2015.06.020","article-title":"A framework for gait-based recognition using Kinect","volume":"68","author":"Kastaniotis","year":"2015","journal-title":"Pattern Recognit. Lett."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1021","DOI":"10.1109\/TSMCB.2010.2043526","article-title":"Action and gait recognition from recovered 3-D human joints","volume":"40","author":"Gu","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part B Cybern."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1111\/1556-4029.13015","article-title":"Reliable Gait Recognition Using 3D Reconstructions and Random Forests\u2013An Anthropometric Approach","volume":"61","author":"Sandau","year":"2016","journal-title":"J. Forensic Sci."},{"key":"ref_18","first-page":"1","article-title":"Viewpoint-independent gait recognition through morphological descriptions of 3D human reconstructions","volume":"48","year":"2016","journal-title":"Image Vis. Comput."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Nizami, I.F., Hong, S., Lee, H., Ahn, S., Toh, K.-A., and Kim, E. (2008, January 3\u20135). Multi-view gait recognition fusion methodology. Proceedings of the 3rd IEEE Conference on Industrial Electronics and Applications, Singapore.","DOI":"10.1109\/ICIEA.2008.4582890"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"966","DOI":"10.1109\/TCSVT.2012.2186744","article-title":"Gait Recognition under Various Viewing Angles Based on Correlated Motion Regression","volume":"22","author":"Kusakunniran","year":"2012","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1109\/TIP.2014.2371335","article-title":"Gait-Based Person Recognition Using Arbitrary View Transformation Model","volume":"24","author":"Muramatsu","year":"2015","journal-title":"IEEE Trans. Image Process."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1602","DOI":"10.1109\/TCYB.2015.2452577","article-title":"View Transformation Model Incorporating Quality Measures for Cross-View Gait Recognition","volume":"46","author":"Muramatsu","year":"2016","journal-title":"IEEE Trans. Cybern."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1016\/j.jvcir.2016.03.020","article-title":"A new approach for multi-view gait recognition on unconstrained paths","volume":"38","year":"2016","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1109\/TPAMI.2016.2545669","article-title":"A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs","volume":"39","author":"Wu","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1016\/j.patcog.2018.06.002","article-title":"Individual identification using a gait dynamics graph","volume":"83","author":"Deng","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1145\/1073204.1073207","article-title":"SCAPE: Shape Completion and Animation of People","volume":"24","author":"Anguelov","year":"2015","journal-title":"ACM Trans. Graph."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bastioni, M., and Simone, R. (2008, January 18\u201320). Ideas and methods for modeling 3D human figures: The principal algorithms used by MakeHuman and their implementation in a new approach to parametric modeling. Proceedings of the 1st ACM Bangalore Annual Conference, COMPUTE 2008, Bangalore, India.","DOI":"10.1145\/1341771.1341782"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1016\/j.patcog.2016.05.030","article-title":"Robust arbitrary view gait recognition based on parametric 3D human body reconstruction and virtual posture synthesis","volume":"60","author":"Luo","year":"2016","journal-title":"Pattern Recognit."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1109\/TIP.2016.2612823","article-title":"Robust Arbitrary-View Gait Recognition based on 3D Partial Similarity Matching","volume":"26","author":"Tang","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_31","unstructured":"CMU (2020, March 15). Carnegie-Mellon Mocap Database. Available online: http:\/\/mocap.cs.cmu.edu."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1109\/TPAMI.2018.2820063","article-title":"Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark","volume":"41","author":"Liang","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Liu, Z., Luo, P., Qiu, S., Wang, X., and Tang, X. (2016, January 27\u201330). DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.124"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"12597","DOI":"10.1007\/s11042-016-3688-4","article-title":"A data-driven editing framework for automatic 3D garment modeling","volume":"76","author":"Liu","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"23","DOI":"10.3389\/fncir.2016.00023","article-title":"Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in Neocortex","volume":"10","author":"Hawkins","year":"2016","journal-title":"Front. Neural Circuits"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Jiang, F., Wang, K., Dong, L., Pan, C., Xu, W., and Yang, K. (2019). Deep Learning Based Joint Resource Scheduling Algorithms for Hybrid MEC Networks. IEEE Internet Things J.","DOI":"10.1109\/JIOT.2019.2954503"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"16094","DOI":"10.1523\/JNEUROSCI.2674-15.2015","article-title":"Selective attention to auditory memory neurally enhances perceptual precision","volume":"35","author":"Lim","year":"2015","journal-title":"J. Neurosci."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Xu, J., Shi, J., Liu, G., Chen, X., and Xu, B. (2018, January 26). Modeling Attention and Memory for Auditory Selection in a Cocktail Party Environment. Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI2018), New Orleans, Louisiana.","DOI":"10.1609\/aaai.v32i1.11879"},{"key":"ref_39","unstructured":"Kaiser, \u0141., Nachum, O., Roy, A., and Bengio, S. (2017, January 24\u201326). Learning to remember rare events. Proceedings of the 5th International Conference on Learning Representations, Toulon, France."},{"key":"ref_40","unstructured":"Gross, R., and Shi, J. (2001). The CMU Motion of Body (MoBo) Database, Robotics Institute, Carnegie Mellon University. Technical Report CMU-RI-TR-01-18."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lee, S., Liu, Y., and Collins, R. (2007, January 18\u201323). Shape variation-based frieze pattern for robust gait recognition. Proceedings of the IEEE Conference on CVPR, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383138"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"3414","DOI":"10.1016\/j.patcog.2012.02.032","article-title":"Silhouette-based gait recognition using Procrustes shape analysis and elliptic Fourier descriptors","volume":"45","author":"Choudhury","year":"2012","journal-title":"Pattern Recognit."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Kusakunniran, W., Wu, Q., Li, H., and Zhang, J. (2009, January 2\u20134). Automatic gait recognition using weighted binary pattern on video. Proceedings of the Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, Genova, Italy.","DOI":"10.1109\/AVSS.2009.44"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zeng, W., and Wang, C. (2013, January 9\u201311). Silhouette-based gait recognition via deterministic learning, Advances in Brain Inspired Cognitive Systems. Proceedings of the 6th International Conference, Beijing, China.","DOI":"10.1007\/978-3-642-38786-9_1"},{"key":"ref_45","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (April, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_46","unstructured":"Yu, S., Tan, D., and Tan, T. (2006, January 20\u201324). A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. Proceedings of the 18th International Conference on Pattern Recognition, Hong Kong, China."},{"key":"ref_47","unstructured":"Kusakunniran, W., Wu, Q., Li, H., and Zhang, J. (October, January 27). Multiple views gait recognition using view transformation model based on optimized gait energy image. Proceedings of the IEEE International Conference Computer Vision, Kyoto, Japan."},{"key":"ref_48","unstructured":"Bashir, K., Xiang, T., and Gong, S. (September, January 31). Cross-view gait recognition using correlation strength. Proceedings of the British Machine Vision Conference, London, UK."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.neucom.2017.02.006","article-title":"Invariant feature extraction for gait recognition using only one uniform model","volume":"239","author":"Yu","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"798","DOI":"10.1016\/j.patcog.2014.09.022","article-title":"Robust view-invariant multiscale gait recognition","volume":"48","author":"Choudhury","year":"2015","journal-title":"Pattern Recognit."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1274","DOI":"10.1109\/TCSVT.2013.2242640","article-title":"Enhanced gabor feature based classification using a regularized locally tensor discriminant model for multiview gait recognition","volume":"23","author":"Hu","year":"2013","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Zheng, S., Zhang, J., Huang, K., He, R., and Tan, T. (2011, January 11\u201314). Robust view transformation model for gait recognition. Proceedings of the 2011 18th IEEE International Conference on Image Processing, Brussels, Belgium.","DOI":"10.1109\/ICIP.2011.6115889"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Makihara, Y., Sagawa, R., Mukaigawa, Y., Echigo, T., and Yagi, Y. (2006, January 7\u201313). Gait recognition using a view transformation model in the frequency domain. Proceedings of the 9th European Conference on Computer Vision, Graz, Austria.","DOI":"10.1007\/11744078_12"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1109\/LSP.2015.2507200","article-title":"Human Body Part Selection by Group Lasso of Motion for Model-Free Gait Recognition","volume":"23","author":"Rida","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_55","first-page":"617","article-title":"Multiview Gait Recognition Based on Patch Distribution Features and Uncorrelated Multilinear Sparse Local Discriminant Canonical Correlation Analysis","volume":"24","author":"Hu","year":"2013","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_56","unstructured":"Hofmann, M., Sural, S., and Rigoll, G. (February, January 31). Gait Recognition in the Presence of Occlusion: A New Dataset and Baseline Algorithms. Proceedings of the 19th International Conferences on Computer Graphics, Visualization and Computer Vision (WSCG), Plzen, Czech Republic."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.patrec.2014.04.004","article-title":"Identification of people walking along curved trajectories","volume":"48","author":"Iwashita","year":"2014","journal-title":"Pattern Recognit. Lett."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Castro, F.M., Mar\u00edn-Jim\u00e9nez, M.J., and Carnicer, R.M. (2014, January 24\u201328). Pyramidal fisher motion for multiview gait recognition. Proceedings of the 22nd International Conference on Pattern Recognition, ICPR 2014, Stockholm, Sweden.","DOI":"10.1109\/ICPR.2014.298"},{"key":"ref_59","unstructured":"Seely, R., Samangooei, S., Lee, M., Carter, J., and Nixon, M. (October, January 29). The University of Southampton Multi-Biometric Tunnel and introducing a novel 3D gait dataset. Proceedings of the 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, Arlington, VA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/6\/1646\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:07:15Z","timestamp":1760173635000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/6\/1646"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,16]]},"references-count":59,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020,3]]}},"alternative-id":["s20061646"],"URL":"https:\/\/doi.org\/10.3390\/s20061646","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,16]]}}}