{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T02:15:51Z","timestamp":1783476951452,"version":"3.55.0"},"reference-count":74,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,2,15]],"date-time":"2023-02-15T00:00:00Z","timestamp":1676419200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Manual annotation for human action recognition with content semantics using 3D Point Cloud (3D-PC) in industrial environments consumes a lot of time and resources. This work aims to recognize, analyze, and model human actions to develop a framework for automatically extracting content semantics. Main Contributions of this work: 1. design a multi-layer structure of various DNN classifiers to detect and extract humans and dynamic objects using 3D-PC preciously, 2. empirical experiments with over 10 subjects for collecting datasets of human actions and activities in one industrial setting, 3. development of an intuitive GUI to verify human actions and its interaction activities with the environment, 4. design and implement a methodology for automatic sequence matching of human actions in 3D-PC. All these procedures are merged in the proposed framework and evaluated in one industrial Use-Case with flexible patch sizes. Comparing the new approach with standard methods has shown that the annotation process can be accelerated by 5.2 times through automation.<\/jats:p>","DOI":"10.3389\/frobt.2023.1028329","type":"journal-article","created":{"date-parts":[[2023,2,15]],"date-time":"2023-02-15T08:59:34Z","timestamp":1676451574000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["A novel approach for automatic annotation of human actions in 3D point clouds for flexible collaborative tasks with industrial robots"],"prefix":"10.3389","volume":"10","author":[{"given":"Sebastian","family":"Krusche","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ibrahim","family":"Al Naser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamad","family":"Bdiwi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steffen","family":"Ihlenfeldt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,2,15]]},"reference":[{"key":"B1","unstructured":"TensorFlow: Large-Scale machine learning on heterogeneous systems\n            AbadiM.\n            AgarwalA.\n            ChenZ.\n            CitroC.\n          2021"},{"key":"B2","unstructured":"Amazon mechanical turk (MTurk)2021"},{"key":"B3","article-title":"PoseTrack: A benchmark for human pose estimation and tracking","author":"Andriluka","year":"2017"},{"key":"B4","unstructured":"Glimpse clouds: Human activity recognition from unstructured feature points\n            BaradelF.\n            WolfC.\n            MilleJ.\n            TaylorG. W.\n          2016"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1711.09561","article-title":"HP-GAN: Probabilistic 3D human motion prediction via gan","author":"Barsoum","year":"2017"},{"key":"B6","first-page":"2500","article-title":"Autonomous disassembly of electric vehicle motors based on robot cognition","author":"Bdiwi","year":"2016"},{"key":"B7","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1016\/j.cviu.2014.06.015","article-title":"An interactive tool for manual, semi-automatic and automatic video annotation","volume":"131","author":"Bianco","year":"2015","journal-title":"Comput. Vis. Image Underst."},{"key":"B8","first-page":"295","article-title":"ViTBAT: Video tracking and behavior annotation tool","author":"Biresaw","year":"2016"},{"key":"B9","volume-title":"The OpenCV library","author":"Bradski","year":"2000"},{"key":"B10","first-page":"2272","article-title":"Exploiting spatial-temporal relationships for 3D pose estimation via graph convolutional networks","author":"Cai","year":"2019"},{"key":"B11","article-title":"Realtime multi-person 2D pose estimation using Part Affinity fields","author":"Cao","year":"2016"},{"key":"B74","unstructured":"OpenPose: Realtime multi-person 2D pose estimation using Part Affinity fields\n            CaoZ.\n            HidalgoG.\n            SimonT.\n            WeiS.-E.\n            SheikhY.\n          2018"},{"key":"B12","article-title":"A short note about kinetics-600","author":"Carreira","year":"2018"},{"key":"B13","article-title":"A short note on the kinetics-700 human action dataset","author":"Carreira","year":"2019"},{"key":"B14","article-title":"HigherHRNet: Scale-Aware representation learning for bottom-up human pose estimation","author":"Cheng","year":"2019"},{"key":"B15","unstructured":"OpenMMLab pose estimation toolbox and benchmark\n            ContributorsM.\n          2021"},{"key":"B16","first-page":"356","article-title":"Open source multipurpose multimedia annotation tool,\u201d in image analysis and recognition","author":"da Silva","year":"2020"},{"key":"B17","unstructured":"Multi-Scale residual graph convolution networks for human motion prediction\n            DangL.\n            NieY.\n            LongC.\n            ZhangQ.\n            LiG.\n            Msr-Gcn\n          2021"},{"key":"B18","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1007\/s00371-015-1066-2","article-title":"A comprehensive survey of human action recognition with spatio-temporal interest point (STIP) detector","volume":"32","author":"Das Dawn","year":"2016","journal-title":"Vis. Comput."},{"key":"B19","first-page":"167","article-title":"Doermann and David Mihalcik, \u201cTools and techniques for video performance evaluation","author":"David","year":"2000"},{"key":"B20","volume-title":"The VGG image annotator (VIA)","author":"Dutta","year":"2019"},{"key":"B21","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2016.213","volume-title":"Convolutional two-stream network fusion for video action recognition","author":"Feichtenhofer","year":"2016"},{"key":"B22","volume-title":"Detecting and recognizing human-object interactions","author":"Gkioxari","year":"2017"},{"key":"B23","first-page":"6047","article-title":"Ava: A video dataset of spatio-temporally localized atomic visual actions","author":"Gu","year":"2018"},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2018.00762","article-title":"DensePose: Dense human pose estimation in the wild","author":"G\u00fcler","year":"2018"},{"key":"B25","doi-asserted-by":"publisher","first-page":"1061","DOI":"10.1007\/s11263-019-01255-4","article-title":"Efficient object annotation via speaking and pointing","volume":"128","author":"Gygli","year":"2020","journal-title":"Int. J. Comput. Vis."},{"key":"B26","doi-asserted-by":"publisher","first-page":"1001955","DOI":"10.3389\/frobt.2022.1001955","article-title":"No-Code robotic programming for agile production: A new markerless-approach for multimodal natural interaction in a human-robot collaboration context","volume":"9","author":"Halim","year":"2022","journal-title":"Front. robotics AI"},{"key":"B27","first-page":"961","article-title":"ActivityNet: A large-scale video benchmark for human activity understanding","author":"Heilbron","year":"2015"},{"key":"B28","article-title":"0MQ - the guide","author":"Hintjens","year":"2011"},{"key":"B29","doi-asserted-by":"publisher","first-page":"2186","DOI":"10.1109\/tpami.2016.2640292","article-title":"Jointly learning heterogeneous features for RGB-D activity recognition","volume":"39","author":"Hu","year":"2017","journal-title":"IEEE Trans. pattern analysis Mach. Intell."},{"key":"B30","unstructured":"Computer vision annotation tool2021"},{"key":"B31","first-page":"10990","article-title":"ETRI-Activity3D: A large-scale RGB-D dataset for robots to recognize daily activities of the elderly","author":"Jang","year":"2020"},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-58545-7_12","article-title":"Whole-body human pose estimation in the wild","author":"Jin","year":"2020"},{"key":"B33","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1038\/nmeth.2281","article-title":"Jaaba: Interactive machine learning for automatic annotation of animal behavior","volume":"10","author":"Kabra","year":"2013","journal-title":"Nat. Methods"},{"key":"B34","article-title":"The kinetics human action video dataset","author":"Kay","year":"2017"},{"key":"B35","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-01252-6_26","article-title":"MultiPoseNet: Fast multi-person pose estimation using pose residual network","author":"Kocabas","year":"2018"},{"key":"B36","article-title":"OpenPifPaf: Composite fields for semantic keypoint detection and spatio-temporal association","author":"Kreiss","year":"2021"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1956","DOI":"10.1007\/s11263-020-01316-z","article-title":"The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale","volume":"128","author":"Kuznetsova","year":"2020","journal-title":"Int. J. Comput. Vis."},{"key":"B38","article-title":"CrowdPose: Efficient crowded scenes pose estimation and A new benchmark","author":"Li","year":"2018"},{"key":"B39","unstructured":"Dynamic multiscale graph neural networks for 3D skeleton-based human motion prediction\n            LiM.\n            ChenS.\n            ZhaoY.\n            ZhangY.\n            WangY.\n            TianQ.\n          2020"},{"key":"B40","first-page":"16261","article-title":"UAV-human: A large benchmark for human behavior understanding with unmanned aerial vehicles","author":"Li","year":"2021"},{"key":"B41","unstructured":"Peeking into the future: Predicting future person activities and locations in videos\n            LiangJ.\n            JiangL.\n            NieblesJ. C.\n            HauptmannA.\n            Fei-FeiL.\n          2019"},{"key":"B42","unstructured":"Microsoft COCO: Common objects in context\n            LinT.-Y.\n            MaireM.\n            HaysJ.\n          2014"},{"key":"B43","doi-asserted-by":"crossref","DOI":"10.1145\/3132734.3132739","article-title":"PKU-MMD: A large scale benchmark for skeleton-based human action understanding","volume":"17","author":"Liu","year":"2017","journal-title":"VSCC"},{"key":"B44","doi-asserted-by":"publisher","first-page":"2684","DOI":"10.1109\/tpami.2019.2916873","article-title":"Ntu RGB+D 120: A large-scale benchmark for 3D human activity understanding","volume":"42","author":"Liu","year":"2020","journal-title":"IEEE Trans. pattern analysis Mach. Intell."},{"key":"B45","article-title":"A short note on the kinetics-700-2020 human action dataset","author":"Lucas","year":"2020"},{"key":"B46","unstructured":"Generating smooth pose sequences for diverse human motion prediction\n            MaoW.\n            LiuM.\n            SalzmannM.\n          2021"},{"key":"B47","unstructured":"History repeats itself: Human motion prediction via motion attention\n            MaoW.\n            LiuM.\n            SalzmannM.\n          2020"},{"key":"B48","unstructured":"Pose transformers (POTR): Human motion prediction with non-autoregressive transformers\n            Mart\u00ednez-Gonz\u00e1lezA.\n            VillamizarM.\n            OdobezJ.-M.\n          2021"},{"key":"B49","volume-title":"Learning to abstract and predict human actions","author":"Morais","year":"2020"},{"key":"B50","unstructured":"Nasdaq Helsinki: QTCOM(Qt group)2021"},{"key":"B51","volume-title":"We don't need no bounding-boxes: Training object class detectors using only human verification","author":"Papadopoulos","year":"2016"},{"key":"B52","first-page":"8024","article-title":"PyTorch: An imperative style, high-performance deep learning library","volume-title":"Advances in neural information processing systems 32","author":"Paszke","year":"2019"},{"key":"B53","unstructured":"Coarse-to-Fine volumetric prediction for single-image 3D human pose\n            PavlakosG.\n            ZhouX.\n            DerpanisK. G.\n            DaniilidisK.\n          2016"},{"key":"B54","first-page":"722","article-title":"Black max planck institute for intelligent systems, and universit\u00e4t konstanz, \u201cBABEL: Bodies, action and behavior with English labels","author":"Punnakkal","year":"2021"},{"key":"B55","doi-asserted-by":"publisher","first-page":"100278","DOI":"10.1016\/j.simpa.2022.100278","article-title":"Havptat: A human activity video pose tracking annotation tool","volume":"12","author":"Quan","year":"2022","journal-title":"Softw. Impacts"},{"key":"B56","first-page":"11179","article-title":"Home action genome: Cooperative compositional action understanding","author":"Rai","year":"2021"},{"key":"B57","first-page":"354","article-title":"Local and global sensors for collision avoidance","author":"Rashid","year":"2020"},{"key":"B58","unstructured":"YOLOv3: An incremental improvement\n            RedmonJ.\n            FarhadiA.\n          2018"},{"key":"B59","article-title":"Mathias lux, vincent charvillat, axel carlier, raynor vliegendhart, and martha larson, \u201cVideoJot: A multifunctional video annotation tool","author":"Riegler","year":"2014"},{"key":"B60","unstructured":"Human motion prediction via spatio-temporal inpainting\n            RuizA. H.\n            GallJ.\n            Moreno-NoguerF.\n          2018"},{"key":"B61","unstructured":"ImageNet large scale visual recognition challenge\n            RussakovskyO.\n            DengJ.\n            SuH.\n            MaS.\n          2014"},{"key":"B62","volume-title":"The visualization toolkit\u2010an object-oriented approach to 3D graphics","author":"Schroeder","year":"2006"},{"key":"B63","first-page":"1010","article-title":"Ntu RGB+D: A large scale dataset for 3D human activity analysis","author":"Shahroudy","year":"2016"},{"key":"B64","first-page":"2613","article-title":"FineGym: A hierarchical video dataset for fine-grained action understanding","author":"Shao","year":"2020"},{"key":"B65","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1109\/TASE.2020.3045655","article-title":"Toward teaching by demonstration for robot-assisted minimally invasive surgery","volume":"18","author":"Su","year":"2021","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"B66","doi-asserted-by":"publisher","first-page":"1564","DOI":"10.1109\/TFUZZ.2022.3157075","article-title":"Fuzzy approximation-based task-space control of robot manipulators with remote center of motion constraint","volume":"30","author":"Su","year":"2022","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"B67","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2019.00584","article-title":"Deep high-resolution representation learning for human pose estimation","author":"Sun","year":"2019"},{"key":"B68","first-page":"411","article-title":"A comprehensive survey on human activity prediction","volume-title":"Lecture notes in computer science, computational science and its applications \u2013 iccsa 2017","author":"Trong","year":"2017"},{"key":"B69","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1007\/s11263-012-0564-1","article-title":"Efficiently scaling up crowdsourced video annotation","volume":"101","author":"Vondrick","year":"2012","journal-title":"Int. J. Comput. Vis."},{"key":"B70","first-page":"2649","article-title":"Cross-view action modeling, learning, and recognition","author":"Wang","year":"2014"},{"key":"B71","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-01231-1_29","article-title":"Simple baselines for human pose estimation and tracking","author":"Xiao","year":"2018"},{"key":"B72","unstructured":"DLow: Diversifying latent flows for diverse human motion prediction\n            YuanY.\n            KitaniK.\n          2020"},{"key":"B73","first-page":"1451","article-title":"LabelMe video: Building a video database with human annotations","author":"Yuen","year":"2009"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1028329\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,15]],"date-time":"2023-02-15T08:59:52Z","timestamp":1676451592000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1028329\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,15]]},"references-count":74,"alternative-id":["10.3389\/frobt.2023.1028329"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1028329","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,15]]},"article-number":"1028329"}}