{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T23:44:26Z","timestamp":1762299866936,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2021,9,13]],"date-time":"2021-09-13T00:00:00Z","timestamp":1631491200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Semantic augmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionised image segmentation and classification, its impact on point cloud is an active research field. In this paper, we propose an instance segmentation and augmentation of 3D point clouds using deep learning architectures. We show the potential of an indirect approach using 2D images and a Mask R-CNN (Region-Based Convolution Neural Network). Our method consists of four core steps. We first project the point cloud onto panoramic 2D images using three types of projections: spherical, cylindrical, and cubic. Next, we homogenise the resulting images to correct the artefacts and the empty pixels to be comparable to images available in common training libraries. These images are then used as input to the Mask R-CNN neural network, designed for 2D instance segmentation. Finally, the obtained predictions are reprojected to the point cloud to obtain the segmentation results. We link the results to a context-aware neural network to augment the semantics. Several tests were performed on different datasets to test the adequacy of the method and its potential for generalisation. The developed algorithm uses only the attributes X, Y, Z, and a projection centre (virtual camera) position as inputs.<\/jats:p>","DOI":"10.3390\/rs13183647","type":"journal-article","created":{"date-parts":[[2021,9,13]],"date-time":"2021-09-13T23:32:23Z","timestamp":1631575943000},"page":"3647","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["3D Point Cloud Semantic Augmentation: Instance Segmentation of 360\u00b0 Panoramas by Deep Learning Techniques"],"prefix":"10.3390","volume":"13","author":[{"given":"Ghizlane","family":"Karara","sequence":"first","affiliation":[{"name":"College of Geomatic Sciences and Surveying Engineering, Institute of Agronomy and Veterinary Medicine,  Rabat 10112, Morocco"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1291-397X","authenticated-orcid":false,"given":"Rafika","family":"Hajji","sequence":"additional","affiliation":[{"name":"College of Geomatic Sciences and Surveying Engineering, Institute of Agronomy and Veterinary Medicine,  Rabat 10112, Morocco"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6368-4399","authenticated-orcid":false,"given":"Florent","family":"Poux","sequence":"additional","affiliation":[{"name":"Geomatics Unit, University of Li\u00e8ge (ULi\u00e8ge), 4000 Li\u00e8ge, Belgium"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,13]]},"reference":[{"key":"ref_1","unstructured":"Poux, F. (2021, May 29). The Future of 3D Point Clouds: A New Perspective, Towards Data Science. Available online: https:\/\/towardsdatascience.com\/the-future-of-3d-point-clouds-a-new-perspective-125b35b558b9."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Poux, F., and Billen, R. (2019). Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised geometric and relationship featuring vs deep learning methods. ISPRS Int. J. Geo-Inf., 8.","DOI":"10.3390\/ijgi8050213"},{"key":"ref_3","unstructured":"He, Y., Yu, H., Liu, X., Yang, Z., Sun, W., Wang, Y., Fu, Q., Zou, Y., and Mian, A. (2021). Deep Learning based 3D Segmentation: A Survey; Deep Learning based 3D Segmentation: A Survey. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Landrieu, L., and Simonovsky, M. (2017). Large-scale Point Cloud Semantic Segmentation with SuperpointGraphs. arXiv.","DOI":"10.1109\/CVPR.2018.00479"},{"key":"ref_5","unstructured":"Topiwala, A. (2021, May 31). Spherical Projection for Point Clouds, Towards Data Science. Available online: https:\/\/towardsdatascience.com\/spherical-projection-for-point-clouds-56a2fc258e6c."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015). Fully convolutional networks for semantic segmentation. arXiv.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2013). Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015). Fast R-CNN. arXiv.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2016). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv.","DOI":"10.1109\/TPAMI.2016.2577031"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017). Mask R-CNN. arXiv.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2015). You Only Look Once: Unified, Real-Time Object Detection. arXiv.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.isprsjprs.2009.06.004","article-title":"Object based image analysis for remote sensing","volume":"65","author":"Blaschke","year":"2010","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Guirado, E., Tabik, S., Alcaraz-Segura, D., Cabello, J., and Herrera, F. (2017). Deep-learning Versus OBIA for scattered shrub detection with Google Earth imagery: Ziziphus lotus as case study. Remote Sens., 9.","DOI":"10.3390\/rs9121220"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"11372","DOI":"10.3390\/rs61111372","article-title":"Object-based land-cover mapping with high resolution aerial photography at a county scale in midwestern USA","volume":"6","author":"Li","year":"2014","journal-title":"Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"12654","DOI":"10.3390\/rs71012654","article-title":"Accuracy optimization for high resolution object-based change detection: An example mapping regional urbanization with 1-m aerial imagery","volume":"7","author":"Pierce","year":"2015","journal-title":"Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Tiede, D., Krafft, P., F\u00fcreder, P., and Lang, S. (2017). Stratified template matching to support refugee camp analysis in OBIA workflows. Remote Sens., 9.","DOI":"10.3390\/rs9040326"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.rse.2004.07.011","article-title":"Object-oriented image analysis for mapping shrub encroachment from 1937 to 2003 in southern New Mexico","volume":"93","author":"Laliberte","year":"2004","journal-title":"Remote Sens. Environ."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"558","DOI":"10.3390\/rs5020558","article-title":"An object-based approach for mapping shrub and tree cover on grassland habitats by use of LiDAR and CIR orthoimages","volume":"5","author":"Hellesen","year":"2013","journal-title":"Remote Sens."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1051","DOI":"10.1016\/j.rse.2007.07.011","article-title":"Monitoring shrubland habitat changes through object-based change identification with airborne multispectral imagery","volume":"112","author":"Stow","year":"2008","journal-title":"Remote Sens. Environ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"355","DOI":"10.5194\/isprs-archives-XLII-2-W17-355-2019","article-title":"Semantic enrichment of point cloud by automatic extraction and enhancement of 360\u00b0 panoramas","volume":"42","author":"Tabkha","year":"2019","journal-title":"Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci."},{"key":"ref_21","unstructured":"Frich, A. (2021, May 31). What Is a Panoramic Photography?. Available online: https:\/\/www.panoramic-photo-guide.com\/panoramic-photography.html."},{"key":"ref_22","unstructured":"Labracherie, R., Num\u00e9rique, F., and Thomas, M. (2021, May 31). La Photographie Panoramique #1: Les Pr\u00e9requis\u2014Les Num\u00e9riques. Available online: https:\/\/www.lesnumeriques.com\/photo\/la-photographie-panoramique-1-les-prerequis-pu100641.html."},{"key":"ref_23","unstructured":"(2021, May 31). La Perspective Conique. Available online: http:\/\/dam.archi.free.fr\/1A1S\/Descriptive\/Cours5.pdf."},{"key":"ref_24","unstructured":"Britannica (2021, May 31). Mercator Projection. Definition, Uses, & Limitations. Available online: https:\/\/www.britannica.com\/science\/Mercator-projection."},{"key":"ref_25","unstructured":"(2021, July 12). Mercator Projection\u2014An Overview, ScienceDirect Topics. Available online: https:\/\/www.sciencedirect.com\/topics\/earth-and-planetary-sciences\/mercator-projection."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1080\/10095020.2015.1017913","article-title":"A study of projections for key point based registration of panoramic terrestrial 3D laser scan","volume":"18","author":"Houshiar","year":"2015","journal-title":"Geo-Spat. Inf. Sci."},{"key":"ref_27","unstructured":"(2021, May 31). Equirectangular Projection\u2014Wikipedia. Available online: https:\/\/en.wikipedia.org\/wiki\/Equirectangular_projection."},{"key":"ref_28","unstructured":"Jepp, P., and Deussen, O. (2010). Pannini: A new projection for rendering wide angle perspective images. Computational Aesthetics in Graphics, Visualization, and Imaging, The Eurographics Association."},{"key":"ref_29","unstructured":"Brown, M. (2015). Content-Aware Projection for Tiny Planets, Eurographics. Short Papers."},{"key":"ref_30","unstructured":"(2021, May 31). Mirror Ball, Angular Map and Spherical. Available online: https:\/\/horo.ch\/docs\/mine\/pdf\/Mb-Am-Sph.pdf."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bello, S.A., Yu, S., and Wang, C. (2020). Review: Deep learning on 3D point clouds. arXiv.","DOI":"10.3390\/rs12111729"},{"key":"ref_32","unstructured":"Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21\u201326). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2437","DOI":"10.1162\/neco.2009.07-08-829","article-title":"Computation with spikes in a winner-take-all network","volume":"21","author":"Oster","year":"2009","journal-title":"Neural Comput."},{"key":"ref_34","first-page":"5100","article-title":"PointNet++: Deep hierarchical feature learning on point sets in a metric space","volume":"2017","author":"Qi","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., and Markham, A. (2019, January 16\u201320). RandLA-Net: Efficient semantic segmentation of large-scale point clouds. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Longbeach, CA, USA.","DOI":"10.1109\/CVPR42600.2020.01112"},{"key":"ref_36","unstructured":"Li, Y., Bu, R., Sun, M., Wu, W., Di, X., and Chen, B. (2018). PointCNN: Convolution On X-Transformed Points. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Thomas, H., Qi, C.R., Deschaud, J.-E., Marcotegui, B., Goulette, F., and Guibas, L.J. (2019, January 27\u201328). KPConv: Flexible and Deformable Convolution for Point Clouds. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00651"},{"key":"ref_38","first-page":"13","article-title":"Dynamic Graph CNN for Learning on Point Clouds","volume":"38","author":"Wang","year":"2018","journal-title":"ACM Trans. Graph."},{"key":"ref_39","unstructured":"Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, L., and Wang, G. (2015). Recent Advances in Convolutional Neural Networks. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.neucom.2019.10.008","article-title":"Impact of fully connected layers on performance of convolutional neural networks for image classification","volume":"378","author":"Basha","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_41","unstructured":"Coursera (2021, May 31). Convolutional Neural Networks. Available online: https:\/\/www.coursera.org\/learn\/convolutional-neural-networks."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1007\/978-3-319-10578-9_23","article-title":"Spatial pyramid pooling in deep convolutional networks for visual recognition","volume":"8691","author":"He","year":"2014","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-319-46448-0_2","article-title":"SSD: Single shot multibox detector","volume":"9905","author":"Liu","year":"2015","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1111\/cgf.14134","article-title":"Fast out-of-core octree generation for massive point clouds","volume":"39","author":"Ohrhallinger","year":"2020","journal-title":"Comput. Graph. Forum"},{"key":"ref_45","unstructured":"Bergounioux, M. (2021, July 20). Quelques M\u00e9thodes de Filtrage en Traitement d\u2019Image. ffhal-00512280v2. Available online: https:\/\/hal.archives-ouvertes.fr\/file\/index\/docid\/569384\/filename\/CoursFiltrage.pdf."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Doll\u00e1r, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2016). Feature Pyramid Networks for Object Detection. arXiv.","DOI":"10.1109\/CVPR.2017.106"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_48","first-page":"740","article-title":"Microsoft COCO: Common objects in context","volume":"Volume 8693 LNCS","author":"Lin","year":"2014","journal-title":"Proceedings of the Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_49","unstructured":"(2021, June 06). Indoor Lidar-RGBD Scan Dataset. Available online: http:\/\/redwood-data.org\/indoor_lidar_rgbd\/."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Zhang, L., Sun, J., and Zheng, Q. (2018). 3D Point Cloud Recognition Based on a Multi-View Convolutional Neural Network. Sensors, 18.","DOI":"10.3390\/s18113681"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Bai, S., Bai, X., Zhou, Z., Zhang, Z., and Latecki, L.J. (2016). GIFT: A Real-time and Scalable 3D Shape Search Engine. arXiv.","DOI":"10.1109\/CVPR.2016.543"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Kalogerakis, E., Averkiou, M., Maji, S., and Chaudhuri, S. (2017, January 21\u201326). 3D Shape Segmentation with Projective Convolutional Networks. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.702"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Qi, C.R., Su, H., Niessner, M., Dai, A., Yan, M., and Guibas, L.J. (2016, January 27\u201330). Volumetric and Multi-View CNNs for Object Classification on 3D Data. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.609"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/18\/3647\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:01:36Z","timestamp":1760166096000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/18\/3647"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,13]]},"references-count":53,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["rs13183647"],"URL":"https:\/\/doi.org\/10.3390\/rs13183647","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2021,9,13]]}}}