{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T16:31:00Z","timestamp":1753893060257,"version":"3.41.2"},"reference-count":69,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,5,10]],"date-time":"2023-05-10T00:00:00Z","timestamp":1683676800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>Combining and completing point cloud data from two or more sensors with arbitrarily relative perspectives in a dynamic, cluttered, and complex environment is challenging, especially when the two sensors have significant perspective differences while the large overlap ratio and feature-rich scene cannot be guaranteed. We create a novel approach targeting this challenging scenario by registering two camera captures in a time series with unknown perspectives and human movements to easily use our system in a real-life scene. In our approach, we first reduce the six unknowns of 3D point cloud completion to three by aligning the ground planes found by our previous perspective-independent 3D ground plane estimation algorithm. Subsequently, we use a histogram-based approach to identify and extract all the humans from each frame generating a three-dimensional (3D) human walking sequence in a time series. To enhance accuracy and performance, we convert 3D human walking sequences to lines by calculating the center of mass (CoM) point of each human body and connecting them. Finally, we match the walking paths in different data trials by minimizing the Fr\u00e9chet distance between two walking paths and using 2D iterative closest point (ICP) to find the remaining three unknowns in the overall transformation matrix for the final alignment. Using this approach, we can successfully register the corresponding walking path of the human between the two cameras\u2019 captures and estimate the transformation matrix between the two sensors.<\/jats:p>","DOI":"10.3389\/frobt.2023.1184614","type":"journal-article","created":{"date-parts":[[2023,5,10]],"date-time":"2023-05-10T20:25:51Z","timestamp":1683750351000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Point cloud completion in challenging indoor scenarios with human motion"],"prefix":"10.3389","volume":"10","author":[{"given":"Chengsi","family":"Zhang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stephen","family":"Czarnuch","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,5,10]]},"reference":[{"key":"B1","first-page":"11753","article-title":"Spinnet: Learning a general surface descriptor for 3d point cloud registration","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Ao","year":"2021"},{"volume-title":"Azure kinect and kinect windows v2 comparison","year":"","key":"B2"},{"volume-title":"Azure kinect dk recorder","year":"2020","key":"B3"},{"volume-title":"D3feat: Joint learning of dense detection and description of 3d local features","year":"2020","author":"Bai","key":"B4"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1103","DOI":"10.1016\/j.acra.2019.08.011","article-title":"Medical 3d printing cost-savings in orthopedic and maxillofacial surgery: Cost analysis of operating room time saved with 3d printed anatomic models and surgical guides","volume":"27","author":"Ballard","year":"2020","journal-title":"Acad. Radiol."},{"key":"B6","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1038\/nrn756","article-title":"The problem of functional localization in the human brain","volume":"3","author":"Brett","year":"2002","journal-title":"Nat. Rev. Neurosci."},{"key":"B7","doi-asserted-by":"publisher","first-page":"1357","DOI":"10.3390\/s22041357","article-title":"Recent advancements in learning algorithms for point clouds: An updated overview","volume":"22","author":"Camuffo","year":"2022","journal-title":"Sensors"},{"key":"B8","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/j.autcon.2017.10.033","article-title":"Performance evaluation of 3d descriptors for object recognition in construction applications","volume":"86","author":"Chen","year":"2018","journal-title":"Automation Constr."},{"key":"B9","doi-asserted-by":"publisher","first-page":"2530","DOI":"10.1109\/tgrs.2019.2952086","article-title":"Plade: A plane-based descriptor for point cloud registration with small overlap","volume":"58","author":"Chen","year":"2019","journal-title":"IEEE Trans. Geoscience Remote Sens."},{"key":"B10","first-page":"410","article-title":"Performance evaluation of 3d keypoints and descriptors","volume-title":"Lecture notes in computer science, proceedings of the international symposium on visual computing","author":"Chen","year":"2016"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.3390\/s18051641","article-title":"Registration of laser scanning point clouds: A review","volume":"18","author":"Cheng","year":"2018","journal-title":"Sensors"},{"key":"B12","first-page":"2514","article-title":"Deep global registration","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Choy","year":"2020"},{"key":"B13","doi-asserted-by":"publisher","first-page":"67179","DOI":"10.1109\/access.2020.2983488","article-title":"3d semantic map construction using improved orb-slam2 for mobile robot in edge computing environment","volume":"8","author":"Cui","year":"2020","journal-title":"IEEE Access"},{"year":"2014","author":"Czarnuch","article-title":"Automated gait analysis in people with Multiple Sclerosis using two unreferenced depth imaging sensors: Preliminary steps","key":"B14"},{"key":"B15","first-page":"3244","article-title":"3d local features for direct pairwise registration","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Deng","year":"2019"},{"key":"B16","first-page":"445","article-title":"Multi-scale clustering of frame-to-frame correspondences for motion segmentation","volume-title":"European conference on computer vision","author":"Dragon","year":"2012"},{"key":"B17","doi-asserted-by":"publisher","first-page":"3724","DOI":"10.3390\/s21113724","article-title":"Automatic super-surface removal in complex 3d indoor environments using iterative region-based ransac","volume":"21","author":"Ebrahimi","year":"2021","journal-title":"Sensors"},{"key":"B18","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1016\/j.inffus.2020.11.002","article-title":"Point-cloud based 3d object detection and classification methods for self-driving applications: A survey and taxonomy","volume":"68","author":"Fernandes","year":"2021","journal-title":"Inf. Fusion"},{"key":"B19","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/978-3-031-16234-3_6","article-title":"Evaluation of point cloud streaming and rendering for vr-based telepresence in the or","volume-title":"International conference on virtual reality and mixed reality","author":"Fischer","year":"2022"},{"key":"B20","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1145\/358669.358692","article-title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography","volume":"24","author":"Fischler","year":"1981","journal-title":"Commun. ACM"},{"key":"B21","doi-asserted-by":"crossref","DOI":"10.1002\/9781119039402","volume-title":"2D and 3D image analysis by moments","author":"Flusser","year":"2016"},{"key":"B22","first-page":"8893","article-title":"Robust point cloud registration framework based on deep graph matching","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Fu","year":"2021"},{"key":"B23","first-page":"5","article-title":"Robust global registration","volume":"2","author":"Gelfand","year":"2005","journal-title":"Symposium geometry Process."},{"key":"B24","first-page":"1759","article-title":"Learning multiview 3d point cloud registration","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Gojcic","year":"2020"},{"key":"B25","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1007\/s11263-015-0824-y","article-title":"A comprehensive performance evaluation of 3d local feature descriptors","volume":"116","author":"Guo","year":"2016","journal-title":"Int. J. Comput. Vis."},{"key":"B26","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1016\/j.cegh.2018.05.006","article-title":"3d scanning applications in medical field: A literature-based review","volume":"7","author":"Haleem","year":"2019","journal-title":"Clin. Epidemiol. Glob. Health"},{"key":"B27","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1007\/s00454-002-2886-1","article-title":"New similarity measures between polylines with applications to morphing and polygon sweeping","volume":"28","author":"Har-Peled","year":"2002","journal-title":"Discrete Comput. Geometry"},{"key":"B28","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1080\/15230406.2014.987694","article-title":"A comparison of usefulness of 2d and 3d representations of urban planning","volume":"42","author":"Herbert","year":"2015","journal-title":"Cartogr. Geogr. Inf. Sci."},{"key":"B29","first-page":"4267","article-title":"Predator: Registration of 3d point clouds with low overlap","volume-title":"Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition","author":"Huang","year":"2021"},{"key":"B30","first-page":"11366","article-title":"Feature-metric registration: A fast semi-supervised approach for robust point cloud registration without correspondences","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Huang","year":"2020"},{"key":"B31","first-page":"7662","article-title":"Pf-net: Point fractal network for 3d point cloud completion","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Huang","year":"2020"},{"key":"B32","doi-asserted-by":"publisher","first-page":"312","DOI":"10.1016\/j.patrec.2016.07.006","article-title":"Good: A global orthographic object descriptor for 3d object recognition and manipulation","volume":"83","author":"Kasaei","year":"2016","journal-title":"Pattern Recognit. Lett."},{"key":"B33","doi-asserted-by":"crossref","first-page":"636","DOI":"10.1109\/ISM.2016.0136","article-title":"Sensing-by-overlaying: A practical implementation of a multiplayer mixed-reality gaming system by integrating a dense point cloud and a real-time camera","volume-title":"2016 IEEE international symposium on multimedia (ISM)","author":"Kurabayashi","year":"2016"},{"key":"B34","first-page":"124","article-title":"Sdrsac: Semidefinite-based randomized approach for robust point cloud registration without correspondences","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Le","year":"2019"},{"key":"B35","doi-asserted-by":"publisher","first-page":"3614","DOI":"10.1109\/tip.2017.2700727","article-title":"Fast descriptors and correspondence propagation for robust global point cloud registration","volume":"26","author":"Lei","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"B36","first-page":"1919","article-title":"End-to-end learning local multi-view descriptors for 3d point clouds","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Li","year":"2020"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1007\/s00521-017-2964-1","article-title":"Scale invariant point feature (sipf) for 3d point clouds and 3d multi-scale object detection","volume":"29","author":"Lin","year":"2018","journal-title":"Neural Comput. Appl."},{"key":"B38","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1061\/9780784482438.021","article-title":"Exploring the potential of image-based 3d geometry and appearance reasoning for automated construction progress monitoring","volume-title":"Computing in civil engineering 2019: Data, sensing, and analytics","author":"Lin","year":"2019"},{"key":"B39","first-page":"17","article-title":"3d point cloud-based visual prediction of icu mobility care activities","volume-title":"Machine learning for healthcare conference","author":"Liu","year":"2018"},{"key":"B40","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1023\/a:1007957421070","article-title":"Robot pose estimation in unknown environments by matching 2d range scans","volume":"18","author":"Lu","year":"1997","journal-title":"J. Intelligent Robotic Syst."},{"key":"B41","first-page":"12","article-title":"Deepvcp: An end-to-end deep neural network for point cloud registration","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"Lu","year":"2019"},{"key":"B42","doi-asserted-by":"crossref","first-page":"1379","DOI":"10.1109\/IROS.2012.6385874","article-title":"Improving generalization for 3d object categorization with global structure histograms","volume-title":"2012 IEEE\/RSJ international conference on intelligent robots and systems","author":"Madry","year":"2012"},{"key":"B43","first-page":"2","article-title":"Fast approximate nearest neighbors with automatic algorithm configuration","volume":"2","author":"Muja","year":"2009","journal-title":"VISAPP"},{"key":"B44","first-page":"184","article-title":"A sift descriptor with global context","volume-title":"2005 IEEE computer society conference on computer vision and pattern recognition (CVPR\u201905)","author":"Mortensen","year":"2005"},{"key":"B45","first-page":"7193","volume-title":"3dregnet: A deep neural network for 3d point registration","author":"Pais","year":"2019"},{"volume-title":"Point primitives for interactive modeling and processing of 3D geometry","year":"2003","author":"Pauly","key":"B46"},{"key":"B47","doi-asserted-by":"publisher","first-page":"115862","DOI":"10.1016\/j.image.2020.115862","article-title":"Point cloud coding: A privileged view driven by a classification taxonomy","volume":"85","author":"Pereira","year":"2020","journal-title":"Signal Process. Image Commun."},{"key":"B48","doi-asserted-by":"publisher","first-page":"741","DOI":"10.3390\/electronics9050741","article-title":"A survey on lidar scanning mechanisms","volume":"9","author":"Raj","year":"2020","journal-title":"Electronics"},{"key":"B49","doi-asserted-by":"crossref","first-page":"3212","DOI":"10.1109\/ROBOT.2009.5152473","article-title":"Fast point feature histograms (fpfh) for 3d registration","volume-title":"2009 IEEE international conference on robotics and automation","author":"Rusu","year":"2009"},{"key":"B50","doi-asserted-by":"crossref","DOI":"10.1109\/ICRA.2011.5980567","article-title":"3D is here: Point cloud library (PCL)","volume-title":"IEEE international conference on robotics and automation (ICRA)","author":"Rusu","year":"2011"},{"key":"B51","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1016\/j.cviu.2014.04.011","article-title":"Shot: Unique signatures of histograms for surface and texture description","volume":"125","author":"Salti","year":"2014","journal-title":"Comput. Vis. Image Underst."},{"volume-title":"Pcrnet: Point cloud registration network using pointnet encoding","year":"2019","author":"Sarode","key":"B52"},{"key":"B53","article-title":"Accuracy estimation for laser point cloud including scanning geometry","volume-title":"Mobile mapping symposium 2007","author":"Schaer","year":"2007"},{"key":"B54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1576246.1531377","article-title":"Curve skeleton extraction from incomplete point cloud","volume-title":"ACM SIGGRAPH 2009 papers","author":"Tagliasacchi","year":"2009"},{"key":"B55","doi-asserted-by":"publisher","first-page":"413","DOI":"10.3390\/s21020413","article-title":"Evaluation of the azure kinect and its comparison to kinect v1 and kinect v2","volume":"21","author":"T\u00f6lgyessy","year":"2021","journal-title":"Sensors"},{"key":"B56","first-page":"356","article-title":"Unique signatures of histograms for local surface description","volume-title":"European conference on computer vision","author":"Tombari","year":"2010"},{"key":"B57","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1109\/FG.2017.67","article-title":"The daily home life activity dataset: A high semantic activity dataset for online recognition","volume-title":"2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017)","author":"Vaquette","year":"2017"},{"key":"B58","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1016\/j.aei.2019.02.007","article-title":"Applications of 3d point cloud data in the construction industry: A fifteen-year review from 2004 to 2018","volume":"39","author":"Wang","year":"2019","journal-title":"Adv. Eng. Inf."},{"key":"B59","first-page":"790","article-title":"Cascaded refinement network for point cloud completion","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Wang","year":"2020"},{"key":"B60","first-page":"1939","article-title":"Point cloud completion by skip-attention network with hierarchical folding","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Wen","year":"2020"},{"key":"B61","doi-asserted-by":"publisher","first-page":"1684","DOI":"10.3390\/s21051684","article-title":"A review: Point cloud-based 3d human joints estimation","volume":"21","author":"Xu","year":"2021","journal-title":"Sensors"},{"key":"B62","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/ICCE.2007.341537","article-title":"3d imaging camera for gaming application","volume-title":"2007 digest of technical papers international conference on consumer electronics","author":"Yahav","year":"2007"},{"key":"B63","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1109\/tro.2020.3033695","article-title":"Teaser: Fast and certifiable point cloud registration","volume":"37","author":"Yang","year":"2020","journal-title":"IEEE Trans. Robotics"},{"key":"B64","first-page":"163","article-title":"A fast and robust local descriptor for 3D point cloud registration","volume-title":"Inf. Sci.","author":"Yang","year":"2016"},{"key":"B65","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1177\/0020294019858217","article-title":"A point cloud registration algorithm based on normal vector and particle swarm optimization","volume":"53","author":"Zhan","year":"2020","journal-title":"Meas. Control"},{"key":"B66","doi-asserted-by":"publisher","first-page":"82 024","DOI":"10.1109\/access.2020.2991346","article-title":"Perspective independent ground plane estimation by 2d and 3d data analysis","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"B67","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/lsens.2018.2889060","article-title":"Real-time human motion behavior detection via cnn using mmwave radar","volume":"3","author":"Zhang","year":"2018","journal-title":"IEEE Sensors Lett."},{"key":"B68","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1016\/j.vrih.2020.05.002","article-title":"Deep learning based point cloud registration: An overview","volume":"2","author":"Zhang","year":"2020","journal-title":"Virtual Real. Intelligent Hardw."},{"key":"B69","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/mmul.2012.24","article-title":"Microsoft kinect sensor and its effect","volume":"19","author":"Zhang","year":"2012","journal-title":"IEEE Multimed."}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1184614\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,10]],"date-time":"2023-05-10T20:26:20Z","timestamp":1683750380000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2023.1184614\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,10]]},"references-count":69,"alternative-id":["10.3389\/frobt.2023.1184614"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2023.1184614","relation":{},"ISSN":["2296-9144"],"issn-type":[{"type":"electronic","value":"2296-9144"}],"subject":[],"published":{"date-parts":[[2023,5,10]]},"article-number":"1184614"}}