{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T06:46:39Z","timestamp":1783147599438,"version":"3.54.6"},"reference-count":73,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T00:00:00Z","timestamp":1692748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:p>Unsupervised Human Action Recognition (U-HAR) methods currently leverage large-scale datasets of human poses to solve this challenging problem. As most of the approaches are dedicated to reaching the best recognition accuracies, no attention has been put into analyzing the resilience of such methods given perturbed data, a likely occurrence in real in-the-wild testing scenarios. Our first contribution is to systematically validate the decrease in performance of current U-HAR state-of-the-art using perturbed or altered data (e.g., obtained by removing some skeletal joints, rotating the entire pose, and injecting geometrical aberrations). Then, we propose a novel framework based on a transformer encoder\u2013decoder with remarkable de-noising capabilities to counter such perturbations effectively. Moreover, we also present additional losses to have robust representations against rotation variances and provide temporal motion consistency. Our model, SKELTER, shows limited drops in performance when skeleton noise is present compared with previous approaches, favoring its use in challenging in-the-wild settings.<\/jats:p>","DOI":"10.3389\/fcomp.2023.1203901","type":"journal-article","created":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T19:38:01Z","timestamp":1692819481000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["SKELTER: unsupervised skeleton action denoising and recognition using transformers"],"prefix":"10.3389","volume":"5","author":[{"given":"Giancarlo","family":"Paoletti","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cigdem","family":"Beyan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessio","family":"Del Bue","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,8,23]]},"reference":[{"key":"B1","first-page":"3","article-title":"Learning local feature descriptors with triplets and shallow convolutional neural networks","author":"Balntas","year":"2016","journal-title":"BMVC"},{"key":"B2","first-page":"2840","article-title":"Coding kendall's shape trajectories for 3D action recognition","author":"Ben Tanfous","year":"2018","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B3","first-page":"4","article-title":"Is space-time attention all you need for video understanding?","author":"Bertasius","year":"2021","journal-title":"ICML"},{"key":"B4","doi-asserted-by":"publisher","first-page":"1070","DOI":"10.1109\/TAFFC.2021.3095425","article-title":"Modeling multiple temporal scales of full-body movements for emotion classification","volume":"14","author":"Beyan","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"B5","first-page":"1877","article-title":"Language models are few-shot learners","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems 33"},{"key":"B6","first-page":"7291","article-title":"Realtime multi-person 2D pose estimation using part affinity fields","author":"Cao","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/GIOTS.2019.8766379","article-title":"Activity recognitions for ambient assisted living using off-the-shelf motion sensing input devices","volume-title":"2019 Global IoT Summit (GIoTS)","author":"Cebanov","year":"2019"},{"key":"B8","first-page":"183","article-title":"Skeleton-based action recognition with shift graph convolutional network","author":"Cheng","year":"2020","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B9","first-page":"1","article-title":"Hierarchical transformer: unsupervised representation learning for skeleton-based human action recognition","volume-title":"2021 IEEE International Conference on Multimedia and Expo (ICME)","author":"Cheng","year":"2021"},{"key":"B10","article-title":"An image is worth 16x16 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2021","journal-title":"ICLR"},{"key":"B11","first-page":"579","article-title":"Skeleton based action recognition with convolutional neural network","volume-title":"2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)","author":"Du","year":""},{"key":"B12","first-page":"1110","article-title":"Hierarchical recurrent neural network for skeleton based action recognition","author":"Du","year":"","journal-title":"Proceedings of IEEE CVPR"},{"key":"B13","doi-asserted-by":"crossref","first-page":"4513","DOI":"10.1109\/ICPR.2014.772","article-title":"Skeletal quads: human action recognition using joint quadruples","volume-title":"2014 22nd International Conference on Pattern Recognition","author":"Evangelidis","year":"2014"},{"key":"B14","first-page":"244","article-title":"Video action transformer network","author":"Girdhar","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B15","first-page":"786","article-title":"Adversarial geometry-aware human motion prediction","author":"Gui","year":"2018","journal-title":"Proceedings of the European Conference on Computer Vision (ECCV)"},{"key":"B16","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1609\/aaai.v36i1.19957","article-title":"Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition","author":"Guo","year":"2022","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B17","article-title":"Benchmarking neural network robustness to common corruptions and perturbations","author":"Hendrycks","year":"2018","journal-title":"International Conference on Learning Representations"},{"key":"B18","first-page":"3288","article-title":"A new representation of skeleton sequences for 3D action recognition","author":"Ke","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B19","doi-asserted-by":"crossref","first-page":"1459","DOI":"10.1109\/WACV.2019.00160","article-title":"Unsupervised feature learning of human actions as trajectories in pose embedding manifold","volume-title":"2019 IEEE Winter Conference on Applications of omputer Vision (WACV)","author":"Kundu","year":"2019"},{"key":"B20","first-page":"597","article-title":"Skeleton-based action recognition with convolutional neural networks","volume-title":"2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)","author":"Li","year":"2017"},{"key":"B21","first-page":"1254","article-title":"Unsupervised learning of view-invariant action representations","volume-title":"32nd Conference on Neural Information Processing Systems, Vol. 2018","author":"Li","year":"2018"},{"key":"B22","first-page":"4741","article-title":"3D human action representation learning via cross-view consistency pursuit","author":"Li","year":"2021","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B23","article-title":"Three-stream convolutional neural network with multi-task and ensemble learning for 3D action recognition","author":"Liang","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops"},{"key":"B24","doi-asserted-by":"crossref","first-page":"2490","DOI":"10.1145\/3394171.3413548","article-title":"MS2L: multi-task self-supervised learning for skeleton based action recognition","author":"Lin","year":"2020","journal-title":"Proceedings of the 28th ACM International Conference on Multimedia"},{"key":"B25","doi-asserted-by":"publisher","first-page":"2684","DOI":"10.1109\/TPAMI.2019.2916873","article-title":"NTU RGB+ D 120: a large-scale benchmark for 3d human activity understanding","volume":"42","author":"Liu","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"B26","first-page":"13","article-title":"ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","volume-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","author":"Lu","year":"2019"},{"key":"B28","first-page":"2891","article-title":"On human motion prediction using recurrent neural networks","author":"Martinez","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B29","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1109\/ICCVW.2011.6130379","article-title":"RGBD-HuDaAct: a color-depth video database for human daily activity recognition","volume-title":"2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)","author":"Ni","year":"2011"},{"key":"B30","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1007\/978-3-030-58529-7_7","article-title":"Unsupervised 3D human pose representation with viewpoint and pose disentanglement","volume-title":"Computer Vision\u2013ECCV 2020: 16th European Conference","author":"Nie","year":"2020"},{"key":"B31","first-page":"465","article-title":"Joint angles similarities and hog2 for action recognition","author":"Ohn-Bar","year":"2013","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops"},{"key":"B32","first-page":"716","article-title":"HON4D: histogram of oriented 4d normals for activity recognition from depth sequences","author":"Oreifej","year":"2013","journal-title":"Proceedings of IEEE CVPR"},{"key":"B33","doi-asserted-by":"publisher","first-page":"131128","DOI":"10.1109\/ACCESS.2022.3229478","article-title":"Graph laplacian-improved convolutional residual autoencoder for unsupervised human action and emotion recognition","volume":"10","author":"Paoletti","year":"2022","journal-title":"IEEE Access"},{"key":"B34","first-page":"6035","article-title":"Subspace clustering for action recognition with covariance representations and temporal pruning","volume-title":"2020 25th International Conference on Pattern Recognition (ICPR)","author":"Paoletti","year":""},{"key":"B35","article-title":"Unsupervised human action recognition with skeletal graph laplacian and self-supervised viewpoints invariance","author":"Paoletti","year":"","journal-title":"32nd British Machine Vision Conference 2021, BMVC 2021"},{"key":"B36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3229097","article-title":"Real-time anomaly detection in elderly behavior with the support of task models","author":"Parvin","year":"2018","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"B37","first-page":"7753","article-title":"3D human pose estimation in video with temporal convolutions and semi-supervised training","author":"Pavllo","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B38","first-page":"10985","article-title":"Action-conditioned 3D human motion synthesis with transformer VAE","author":"Petrovich","year":"2021","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B39","first-page":"99","article-title":"IoT-powered monitoring systems for geriatric healthcare: overview","author":"Petrushin","year":"2022","journal-title":"Internet of Things for Human-Centered Design: Application to Elderly Healthcare"},{"key":"B40","doi-asserted-by":"publisher","first-page":"976","DOI":"10.1016\/j.imavis.2009.11.014","article-title":"A survey on vision-based human action recognition","volume":"28","author":"Poppe","year":"2010","journal-title":"Image Vis. Comput."},{"key":"B41","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.patcog.2015.11.019","article-title":"3D skeleton-based human action classification: a survey","volume":"53","author":"Presti","year":"2016","journal-title":"Pattern Recogn."},{"key":"B42","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.ins.2021.04.023","article-title":"Augmented skeleton based contrastive action learning with momentum LSTM for unsupervised action recognition","volume":"569","author":"Rao","year":"2021","journal-title":"Inform. Sci."},{"key":"B43","article-title":"ImageNet-21k pretraining for the masses","author":"Ridnik","year":"2021","journal-title":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)"},{"key":"B44","first-page":"1010","article-title":"NTU RGB+ D: a large scale dataset for 3D human activity analysis","author":"Shahroudy","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B45","first-page":"7912","article-title":"Skeleton-based action recognition with directed graph neural networks","author":"Shi","year":"","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B46","first-page":"12026","article-title":"Two-stream adaptive graph convolutional networks for skeleton-based action recognition","author":"Shi","year":"","journal-title":"Proceedings of IEEE CVPR"},{"key":"B47","first-page":"1227","article-title":"An attention enhanced graph convolutional LSTM network for skeleton-based action recognition","author":"Si","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B48","doi-asserted-by":"publisher","first-page":"3459","DOI":"10.1109\/TIP.2018.2818328","article-title":"Spatio-temporal attention-based LSTM networks for 3D action recognition and detection","volume":"27","author":"Song","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"B49","first-page":"9631","article-title":"Predict & cluster: unsupervised skeleton based action recognition","author":"Su","year":"","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B50","article-title":"Vl-BERT: pre-training of generic visual-linguistic representations","author":"Su","year":"","journal-title":"International Conference on Learning Representations"},{"key":"B51","doi-asserted-by":"crossref","first-page":"1655","DOI":"10.1145\/3474085.3475307","article-title":"Skeleton-contrastive 3D action representation learning","author":"Thoker","year":"2021","journal-title":"Proceedings of the 29th ACM International Conference on Multimedia"},{"key":"B52","first-page":"1938","article-title":"Minimax estimation of maximum mean discrepancy with radial kernels","author":"Tolstikhin","year":"2016","journal-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems"},{"key":"B53","first-page":"6000","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems"},{"key":"B54","first-page":"588","article-title":"Human action recognition by representing 3D skeletons as points in a lie group","author":"Vemulapalli","year":"2014","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B55","first-page":"4471","article-title":"Rolling rotations for recognizing human actions from 3D skeletal data","author":"Vemulapalli","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B56","doi-asserted-by":"crossref","first-page":"1290","DOI":"10.1109\/CVPR.2012.6247813","article-title":"Mining actionlet ensemble for action recognition with depth cameras","volume-title":"2012 IEEE Conference on Computer Vision and Pattern Recognition","author":"Wang","year":"2012"},{"key":"B57","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1109\/TIP.2019.2925285","article-title":"A comparative review of recent kinect-based action recognition algorithms","volume":"29","author":"Wang","year":"","journal-title":"IEEE Trans. Image Process."},{"key":"B58","doi-asserted-by":"publisher","first-page":"6940","DOI":"10.1109\/TPAMI.2020.3032738","article-title":"Learning multi-view interactional skeleton graph for action recognition","volume":"45","author":"Wang","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B59","first-page":"1810","article-title":"Learning deep transformer models for machine translation","author":"Wang","year":"","journal-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"},{"key":"B60","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1049\/cit2.12014","article-title":"Deep learning-based action recognition with 3D skeleton: a survey","volume":"6","author":"Xing","year":"2021","journal-title":"CAAI Trans. Intell. Technol."},{"key":"B61","first-page":"899","article-title":"Deep kinematics analysis for monocular 3D human pose estimation","author":"Xu","year":"","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B62","first-page":"624","article-title":"Prototypical contrast and reverse prediction: Unsupervised skeleton based action recognition","volume-title":"IEEE Transactions on Multimedia, Vol. 25","author":"Xu","year":"2023"},{"key":"B63","doi-asserted-by":"publisher","first-page":"15990","DOI":"10.1109\/JIOT.2020.3042986","article-title":"Attention-based multilevel co-occurrence graph convolutional LSTM for 3-D action recognition","volume":"8","author":"Xu","year":"","journal-title":"IEEE Internet Things J."},{"key":"B64","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.12328","article-title":"Spatial temporal graph convolutional networks for skeleton-based action recognition","author":"Yan","year":"2018","journal-title":"Thirty-Second AAAI Conference on Artificial Intelligence"},{"key":"B65","first-page":"804","article-title":"Super normal vector for activity recognition using depth sequences","author":"Yang","year":"2014","journal-title":"Proceedings of the IEEE Conference on omputer Vision and Pattern Recognition"},{"key":"B66","first-page":"5753","article-title":"XLNet: generalized autoregressive pretraining for language understanding","volume-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","author":"Yang","year":"2019"},{"key":"B67","article-title":"Benchmarking the robustness of spatial-temporal models against corruptions","author":"Yi","year":"2021","journal-title":"Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)"},{"key":"B68","first-page":"2752","article-title":"The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection","volume-title":"Proceedings of IEEE ICCV","author":"Zanfir","year":"2013"},{"key":"B69","first-page":"2117","article-title":"View adaptive recurrent neural networks for high performance human action recognition from skeleton data","author":"Zhang","year":"2017","journal-title":"Proceedings of the IEEE international conference on computer vision"},{"key":"B70","doi-asserted-by":"crossref","first-page":"1963","DOI":"10.1109\/TPAMI.2019.2896631","article-title":"View adaptive neural networks for high performance skeleton-based human action recognition","volume":"41","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"B71","first-page":"1112","article-title":"Semantics-guided neural networks for efficient skeleton-based human action recognition","author":"Zhang","year":"","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B72","first-page":"14333","article-title":"Context aware graph convolution for skeleton-based action recognition","author":"Zhang","year":"","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B73","first-page":"11656","article-title":"3D human pose estimation with spatial and temporal transformers","author":"Zheng","year":"2021","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B74","article-title":"Unsupervised representation learning with long-term dynamics for skeleton based action recognition","author":"Zheng","year":"2018","journal-title":"AAAI Conference on Artificial Intelligence"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2023.1203901\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T19:38:34Z","timestamp":1692819514000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2023.1203901\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,23]]},"references-count":73,"alternative-id":["10.3389\/fcomp.2023.1203901"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2023.1203901","relation":{},"ISSN":["2624-9898"],"issn-type":[{"value":"2624-9898","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,23]]},"article-number":"1203901"}}