{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T10:29:45Z","timestamp":1763202585255,"version":"build-2065373602"},"reference-count":67,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This work proposes a novel unsupervised self-organizing network, called the Self-Organizing Convolutional Echo State Network (SO-ConvESN), for learning node centroids and interconnectivity maps compatible with the deterministic initialization of Echo State Network (ESN) input and reservoir weights, in the context of human action recognition (HAR). To ensure stability and echo state property in the reservoir, Recurrent Plots (RPs) and Recurrence Quantification Analysis (RQA) techniques are exploited for explainability and characterization of the reservoir dynamics and hence tuning ESN hyperparameters. The optimized self-organizing reservoirs are cascaded with a Convolutional Neural Network (CNN) to ensure that the activation of internal echo state representations (ESRs) echoes similar topological qualities and temporal features of the input time-series, and the CNN efficiently learns the dynamics and multiscale temporal features from the ESRs for action recognition. The hyperparameter optimization (HPO) algorithms are additionally adopted to optimize the CNN stage in SO-ConvESN. Experimental results on the HAR problem using several publicly available 3D-skeleton-based action datasets demonstrate the showcasing of the RPs and RQA technique in examining the explainability of reservoir dynamics for designing stable self-organizing reservoirs and the usefulness of implementing HPOs in SO-ConvESN for the HAR task. The proposed SO-ConvESN exhibits competitive recognition accuracy.<\/jats:p>","DOI":"10.3390\/s22051905","type":"journal-article","created":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T21:25:14Z","timestamp":1646169914000},"page":"1905","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["On the Post Hoc Explainability of Optimized Self-Organizing Reservoir Network for Action Recognition"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1050-6238","authenticated-orcid":false,"given":"Gin Chong","family":"Lee","sequence":"first","affiliation":[{"name":"Faculty of Engineering and Technology, Multimedia University, Jalan Ayer Keroh Lama, Melaka 75450, Malaysia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7867-2665","authenticated-orcid":false,"given":"Chu Kiong","family":"Loo","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence, Faculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur 50603, Malaysia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Ma, X., Song, R., Rong, X., Tian, X., Tian, G., and Li, Y. (2017, January 20\u201322). Deep learning based human action recognition: A survey. Proceedings of the 2017 Chinese Automation Congress (CAC), Jinan, China.","DOI":"10.1109\/CAC.2017.8243438"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Cichy, R.M., Khosla, A., Pantazis, D., Torralba, A., and Oliva, A. (2016). Deep neural networks predict hierarchical spatio-temporal cortical dynamics of human visual object recognition. arXiv.","DOI":"10.1038\/srep27755"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.eswa.2016.06.011","article-title":"A survey on using domain and contextual knowledge for human activity recognition in video streams","volume":"63","author":"Onofri","year":"2016","journal-title":"Expert Syst. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1016\/j.patcog.2015.11.019","article-title":"3D Skeleton-based Human Action Classification: A Survey","volume":"53","author":"Presti","year":"2016","journal-title":"Pattern Recognit."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Papadopoulos, G.T., Axenopoulos, A., and Daras, P. (2014, January 6\u201310). Real-time skeleton-tracking-based human action recognition using Kinect data. Proceedings of the 20th Anniversary International Conference on MultiMedia Modeling, Dublin, Ireland.","DOI":"10.1007\/978-3-319-04114-8_40"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Huang, J.D. (2011, January 24\u201326). Kinerehab: A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Proceedings of the 13th international ACM SIGACCESS Conference on Computers and Accessibility, Dundee, UK.","DOI":"10.1145\/2049536.2049627"},{"key":"ref_7","unstructured":"Kong, Y., and Fu, Y. (2018). Human action recognition and prediction: A survey. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. (2011, January 20\u201325). Real-time human pose recognition in parts from single depth images. Proceedings of the CVPR 2011, Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995316"},{"key":"ref_9","first-page":"172","article-title":"The \u201cecho state\u201d approach to analysing and training recurrent neural networks-with an erratum note","volume":"148","author":"Jaeger","year":"2001","journal-title":"Tech. Rep. Ger. Natl. Res. Cent. Inform. Technol. GMD"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ma, Q., Shen, L., Chen, E., Tian, S., Wang, J., and Cottrell, G.W. (2017, January 19\u201325). WALKING WALKing walking: Action Recognition from Action Echoes. Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/342"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_12","unstructured":"Wu, Q., Fokoue, E.P., and Kudithipudi, D. (2018). On the statistical challenges of echo state networks and some potential remedies. arXiv."},{"key":"ref_13","unstructured":"Mici, L., Hinaut, X., and Wermter, S. (2016, January 27\u201329). Activity recognition with echo state networks using 3D body joints and objects category. Proceedings of the European Symposium On Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium."},{"key":"ref_14","unstructured":"Kingma, D., and Lei-Ba, J. (2015). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bochinski, E., Senst, T., and Sikora, T. (2017, January 17\u201320). Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.","DOI":"10.1109\/ICIP.2017.8297018"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1111\/j.1551-6708.1987.tb00862.x","article-title":"Competitive learning: From interactive activation to adaptive resonance","volume":"11","author":"Grossberg","year":"1987","journal-title":"Cogn. Sci."},{"key":"ref_17","unstructured":"Jockusch, J., and Ritter, H. (1999, January 10\u201316). An instantaneous topological mapping model for correlated stimuli. Proceedings of the International Joint Conference on Neural Networks, Washington, DC, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1109\/TNNLS.2016.2630802","article-title":"Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis","volume":"29","author":"Bianchi","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1111\/1467-7687.00104","article-title":"Neural plasticity and human development: The role of early experience in sculpting memory systems","volume":"3","author":"Nelson","year":"2000","journal-title":"Dev. Sci."},{"key":"ref_20","unstructured":"Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M.M.A., and Adams, R.P. (2015, January 7\u20139). Scalable Bayesian Optimization Using Deep Neural Networks. Proceedings of the 32nd International Conference on International Conference on Machine Learning, Lille, France."},{"key":"ref_21","unstructured":"Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., and Talwalkar, A. (2020). A System for Massively Parallel Hyperparameter Tuning. arXiv."},{"key":"ref_22","unstructured":"Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W.M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., and Simonyan, K. (2017). Population Based Training of Neural Networks. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Biswas, K., and Basu, S.K. (2011, January 6\u20138). Gesture recognition using Microsoft Kinect. Proceedings of the 5th International Conference on Automation, Robotics and Applications, Wellington, New Zealand.","DOI":"10.1109\/ICARA.2011.6144864"},{"key":"ref_24","unstructured":"Bhattacharya, S., Czejdo, B., and Perez, N. (December, January 29). Gesture classification with machine learning using Knect sensor data. Proceedings of the 3rd International Conference on Emerging Applications of Information Technology, Kolkata, India."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Patsadu, O., Nukoolkit, C., and Watanapa, B. (June, January 30). Human gesture recognition using Kinect camera. Proceedings of the 2012 Ninth International Conference on Computer Science and Software Engineering (JCSSE), Bangkok, Thailand.","DOI":"10.1109\/JCSSE.2012.6261920"},{"key":"ref_26","first-page":"157","article-title":"Coarse-dtw for sparse time series alignment","volume":"Volume 9785","author":"Vilar","year":"2016","journal-title":"Advanced Analysis and Learning on Temporal Data"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.advengsoft.2014.07.005","article-title":"Easy gesture recognition for Kinect","volume":"76","author":"Ibanez","year":"2014","journal-title":"Adv. Eng. Softw."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Marteau, P.F., Gibet, S., and Reverdy, C. (2014, January 24\u201328). Down-sampling coupled to elastic kernel machines for efficient recognition of isolated gestures. Proceedings of the 22nd International Conference on Pattern Recognition, Stockholm, Sweden.","DOI":"10.1109\/ICPR.2014.71"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lev, G., Sadeh, G., Klein, B., and Wolf, L. (2016). RNN Fisher Vectors for Action Recognition and Image Annotation. arXiv.","DOI":"10.1007\/978-3-319-46466-4_50"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1963","DOI":"10.1109\/TPAMI.2019.2896631","article-title":"View adaptive neural networks for high performance skeleton-based human action recognition","volume":"41","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","unstructured":"Gallicchio, C., and Micheli, A. (2016, January 28). A reservoir computing approach for human gesture recognition from Kinect data. Proceedings of the Workshop Artificial Intelligence for Ambient Assisted Living, Genova, Italy."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/j.neunet.2019.05.008","article-title":"Time series classification with Echo Memory Networks","volume":"117","author":"Ma","year":"2019","journal-title":"Neural Netw."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Bengio, Y. (2013, January 29\u201331). Deep learning of representations: Looking forward. Proceedings of the International Conference on Statistical Language and Speech Processing, Tarragona, Spain.","DOI":"10.1007\/978-3-642-39593-2_1"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.eswa.2018.03.056","article-title":"Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges","volume":"105","author":"Nweke","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep learning for sensor-based activity recognition: A survey","volume":"119","author":"Wang","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_36","unstructured":"Palangi, H., Deng, L., and Ward, R. (2013). Learning Input and Recurrent Weight Matrices in Echo State Networks. arXiv."},{"key":"ref_37","unstructured":"Arrieta, A.B., Gil-Lopez, S., La\u00f1a, I., Bilbao, M.N., and Ser, J.D. (2021). On the Post-hoc Explainability of Deep Echo State Networks for Time Series Forecasting, Image and Video Classification. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1016\/j.neucom.2014.01.036","article-title":"Self-organization and lateral interaction in echo state network reservoirs","volume":"138","author":"Boccato","year":"2014","journal-title":"Neurocomputing"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/BF00337288","article-title":"Self-organized formation of topologically correct feature maps","volume":"43","author":"Kohonen","year":"2004","journal-title":"Biol. Cybern."},{"key":"ref_40","unstructured":"Fritzke, B. (1994, January 1). A Growing Neural Gas Network Learns Topologies. Proceedings of the 7th International Conference on Neural Information Processing Systems."},{"key":"ref_41","unstructured":"Martinetz, T.M., and Schulten, K.J. (1991, January 24\u201328). A \u201cNeural Gas\u201d Network Learns Topologies. Proceedings of the International Conference on Artificial Neural Networks, Espoo, Finland."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1441","DOI":"10.1016\/0893-6080(94)90091-4","article-title":"Growing cell structures: A self-organizing network for unsupervised and supervised learnin","volume":"7","author":"Fritzke","year":"1994","journal-title":"Neural Netw."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1109\/2.33","article-title":"The ART of adaptive pattern recognition by a self-organizing neural network","volume":"3","author":"Carpenter","year":"1988","journal-title":"Computer"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/j.physrep.2006.11.001","article-title":"Recurrence plots for the analysis of complex systems","volume":"438","author":"Marwan","year":"2007","journal-title":"Phys. Rep."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1142\/S0218127411029008","article-title":"How to Avoid Potential Pitfalls in Recurrence Plot Based Data Analysis","volume":"21","author":"Marwan","year":"2011","journal-title":"Int. J. Bifurc. Chaos"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"042919","DOI":"10.1103\/PhysRevE.90.042919","article-title":"Entropy of weighted recurrence plots","volume":"90","author":"Eroglu","year":"2014","journal-title":"Phys. Rev. E"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1016\/j.neucom.2016.12.089","article-title":"Deep reservoir computing: A critical experimental analysis","volume":"268","author":"Gallicchio","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Xie, L., and Yuille, A.L. (2017, January 22\u201329). Genetic CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.154"},{"key":"ref_49","first-page":"281","article-title":"Random Search for Hyper-Parameter Optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_50","unstructured":"Snoek, J., Larochelle, H., and Adams, R.P. (2012, January 3\u20136). Practical Bayesian Optimization of Machine Learning Algorithms. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, CA, USA."},{"key":"ref_51","unstructured":"Domhan, T., Springenberg, J.T., and Hutter, F. (2015, January 25\u201331). Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves. Proceedings of the IJCAI International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina."},{"key":"ref_52","unstructured":"Eggensperger, K. (2013, January 10). Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters. Proceedings of the NIPS Workshop on Bayesian Optimization in Theory and Practice, Lake Tahoe, NV, USA."},{"key":"ref_53","unstructured":"Jamieson, K.G., and Talwalkar, A.S. (2016). Non-stochastic Best Arm Identification and Hyperparameter Optimization. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1016\/j.neunet.2012.09.012","article-title":"Noise-enhanced clustering and competitive learning algorithm","volume":"37","author":"Osoba","year":"2013","journal-title":"Neural Netw."},{"key":"ref_55","first-page":"2493","article-title":"Natural Language Processing (Almost) from Scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_56","unstructured":"Nair, V., and Hinto, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machine. Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Li, W., Zhang, Z., and Liu, Z. (2010, January 13\u201318). Action recognition based on a bag of 3d point. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition\u2014Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543273"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Seidenari, L., Varano, V., Berretti, S., Bimbo, A.D., and Pala, P. (2013, January 23\u201328). Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Portland, OR, USA.","DOI":"10.1109\/CVPRW.2013.77"},{"key":"ref_59","unstructured":"Antunes, J., Bernardino, A., Smailagic, A., and Siewiorek, D. (2018, January 3\u20136). AHA-3D: A Labelled Dataset for Senior Fitness Exercise Recognition and Segmentation from 3D Skeletal Data. Proceedings of the Vision International Behaviour Understanding Workshop, British Machine Vision Conference, Newcastle upon Tyne, UK."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Zhang, X., Wang, Y., Gou, M., Sznaier, M., and Camps, O. (2016, January 27\u201330). Efficient temporal sequence comparison and classification using gram matrix embeddings on a riemannian manifold. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.487"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks Trade, Springer.","DOI":"10.1007\/978-3-642-35289-8_26"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"1906","DOI":"10.1021\/ac60319a045","article-title":"Smoothing and differentiation of data by simplified least square procedure","volume":"44","author":"Steinier","year":"1972","journal-title":"Anal. Chem."},{"key":"ref_63","unstructured":"Hussein, M.E., Torki, M., Gowayyed, M.A., and El-Saban, M. (2013, January 3\u20139). Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations. Proceedings of the Twenty-third international joint conference on artificial intelligence, Beijing, China."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Vemulapalli, R., Arrate, F., and Chellappa, R. (2014, January 23\u201328). Human action recognition by representing 3d skeletons as points in a lie group. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.82"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.imavis.2015.09.007","article-title":"Hankelet-based dynamical systems modeling for 3d action recognition","volume":"44","author":"Presti","year":"2015","journal-title":"Image Vis. Comput."},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Gao, X., Hu, W., Tang, J., Liu, J., and Guo, Z. (2019). Optimized Skeleton-based Action Recognition via Sparsified Graph Regression. arXiv.","DOI":"10.1145\/3343031.3351170"},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Li, C., Cui, Z., Zheng, W., Xu, C., and Yang, J. (2018). Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition. arXiv.","DOI":"10.1609\/aaai.v32i1.11776"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1905\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:29:57Z","timestamp":1760135397000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/5\/1905"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,1]]},"references-count":67,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22051905"],"URL":"https:\/\/doi.org\/10.3390\/s22051905","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,3,1]]}}}