{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T13:22:07Z","timestamp":1768828927758,"version":"3.49.0"},"reference-count":55,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T00:00:00Z","timestamp":1669680000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union\u2019s Horizon 2020 research and innovation programme under the Marie Sk\u0142odowska-Curie grant agreement","award":["860843"],"award-info":[{"award-number":["860843"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Currently, the use of 3D point clouds is rapidly increasing in many engineering fields, such as geoscience and manufacturing. Various studies have developed intelligent segmentation models providing accurate results, while only a few of them provide additional insights into the efficiency and robustness of their proposed models. The process of segmentation in the image domain has been studied to a great extent and the research findings are tremendous. However, the segmentation analysis with point clouds is considered particularly challenging due to their unordered and irregular nature. Additionally, solving downstream tasks with 3D point clouds is computationally inefficient, as point clouds normally consist of thousands or millions of points sparsely distributed in 3D space. Thus, there is a significant need for rigorous evaluation of the design characteristics of segmentation models, to be effective and practical. Consequently, in this paper, an in-depth analysis of five fundamental and representative deep learning models for 3D point cloud segmentation is presented. Specifically, we investigate multiple experimental dimensions, such as accuracy, efficiency, and robustness in part segmentation (ShapeNet) and scene segmentation (S3DIS), to assess the effective utilization of the models. Moreover, we create a correspondence between their design properties and experimental properties. For example, we show that convolution-based models that incorporate adaptive weight or position pooling local aggregation operations achieve superior accuracy and robustness to point-wise MLPs, while the latter ones show higher efficiency in time and memory allocation. Our findings pave the way for an effective 3D point cloud segmentation model selection and enlighten the research on point clouds and deep learning.<\/jats:p>","DOI":"10.3390\/rs14236049","type":"journal-article","created":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T05:45:22Z","timestamp":1669787122000},"page":"6049","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Rethinking Design and Evaluation of 3D Point Cloud Segmentation Models"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3736-1155","authenticated-orcid":false,"given":"Thanasis","family":"Zoumpekas","sequence":"first","affiliation":[{"name":"WAI Research Group, Department of Mathematics and Computer Science, University of Barcelona, 08007 Barcelona, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1939-8963","authenticated-orcid":false,"given":"Maria","family":"Salam\u00f3","sequence":"additional","affiliation":[{"name":"WAI Research Group, Department of Mathematics and Computer Science, University of Barcelona, 08007 Barcelona, Spain"},{"name":"UBICS Institute, University of Barcelona, 08007 Barcelona, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2184-2800","authenticated-orcid":false,"given":"Anna","family":"Puig","sequence":"additional","affiliation":[{"name":"WAI Research Group, Department of Mathematics and Computer Science, University of Barcelona, 08007 Barcelona, Spain"},{"name":"IMUB Institute, University of Barcelona, 08007 Barcelona, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1951","DOI":"10.1109\/TITS.2019.2909066","article-title":"Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving","volume":"21","author":"Zhou","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"722","DOI":"10.1109\/TITS.2020.3023541","article-title":"Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review","volume":"23","author":"Cui","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Bello, S.A., Yu, S., Wang, C., Adam, J.M., and Li, J. (2020). Review: Deep Learning on 3D Point Clouds. Remote Sens., 12.","DOI":"10.3390\/rs12111729"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5702815","DOI":"10.1109\/TGRS.2022.3161982","article-title":"Semantic Segmentation of Terrestrial Laser Scanning Point Clouds Using Locally Enhanced Image-Based Geometric Representations","volume":"60","author":"Cai","year":"2022","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"104250","DOI":"10.1016\/j.autcon.2022.104250","article-title":"Automatic region-growing system for the segmentation of large point clouds","volume":"138","author":"Poux","year":"2022","journal-title":"Autom. Constr."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1007\/s11831-019-09320-4","article-title":"Computational Methods of Acquisition and Processing of 3D Point Cloud Data for Construction Applications","volume":"27","author":"Wang","year":"2020","journal-title":"Arch. Comput. Methods Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Grilli, E., and Remondino, F. (2019). Classification of 3D Digital Heritage. Remote Sens., 11.","DOI":"10.3390\/rs11070847"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/j.cag.2022.06.012","article-title":"Multimodal registration across 3D point clouds and CT-volumes","volume":"106","author":"Saiti","year":"2022","journal-title":"Comput. Graph."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"4338","DOI":"10.1109\/TPAMI.2020.3005434","article-title":"Deep Learning for 3D Point Clouds: A Survey","volume":"43","author":"Guo","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Nguyen, A., and Le, B. (2013, January 12\u201315). 3D point cloud segmentation: A survey. Proceedings of the 6th IEEE Conference on Robotics, Automation and Mechatronics (RAM), Manila, Philippines.","DOI":"10.1109\/RAM.2013.6758588"},{"key":"ref_11","unstructured":"Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21\u201326). PointNet: Deep learning on point sets for 3D classification and segmentation. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA."},{"key":"ref_12","unstructured":"Qi, C.R., Yi, L., Su, H., Guibas, L.J., Li, C.R.Q., Hao, Y., Leonidas, S., and Guibas, J. (2017, January 4\u20139). PointNet++: Deep hierarchical feature learning on point sets in a metric space. Proceedings of the Advances in Neural Information Processing Systems. Neural Information Processing Systems Foundation, Long Beach, CA, USA."},{"key":"ref_13","unstructured":"Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., and Guibas, L. (November, January 27). KPConv: Flexible and deformable convolution for point clouds. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Republic of Korea."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, Z., Hu, H., Cao, Y., Zhang, Z., and Tong, X. (2020, January 23\u201328). A Closer Look at Local Aggregation Operators in Point Cloud Analysis. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK. LNCS.","DOI":"10.1007\/978-3-030-58592-1_20"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Liu, Y., Fan, B., Xiang, S., and Pan, C. (2019, January 16\u201320). Relation-Shape Convolutional Neural Network for Point Cloud Analysis. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00910"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2980179.2980238","article-title":"A scalable active framework for region annotation in 3D shape collections","volume":"35","author":"Yi","year":"2016","journal-title":"ACM Trans. Graph."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Armeni, I., Sener, O., Zamir, A.R., Jiang, H., Brilakis, I., Fischer, M., and Savarese, S. (2016, January 27\u201330). 3D semantic parsing of large-scale indoor spaces. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.170"},{"key":"ref_18","first-page":"3523","article-title":"Image Segmentation Using Deep Learning: A Survey","volume":"44","author":"Minaee","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/s10278-017-9983-4","article-title":"Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions","volume":"30","author":"Akkus","year":"2017","journal-title":"J. Digit. Imaging"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1182","DOI":"10.1109\/TMI.2016.2528129","article-title":"Automatic Detection of Cerebral Microbleeds from MR Images via 3D Convolutional Neural Networks","volume":"35","author":"Dou","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.asoc.2018.05.018","article-title":"A survey on deep learning techniques for image and video semantic segmentation","volume":"70","author":"Oprea","year":"2018","journal-title":"Appl. Soft Comput."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1007\/s11263-020-01383-2","article-title":"Benchmarking the Robustness of Semantic Segmentation Models with Respect to Common Corruptions","volume":"129","author":"Kamann","year":"2020","journal-title":"Int. J. Comput. Vis."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kamann, C., and Rother, C. (2020, January 23\u201328). Increasing the Robustness of Semantic Segmentation Models with Painting-by-Numbers. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK. LNCS.","DOI":"10.1007\/978-3-030-58607-2_22"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","article-title":"nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation","volume":"18","author":"Isensee","year":"2020","journal-title":"Nat. Methods"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1016\/j.isprsjprs.2015.01.016","article-title":"Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers","volume":"105","author":"Weinmann","year":"2015","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/MGRS.2019.2937630","article-title":"Linking Points with Labels in 3D: A Review of Point Cloud Semantic Segmentation","volume":"8","author":"Xie","year":"2020","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"179118","DOI":"10.1109\/ACCESS.2019.2958671","article-title":"A Review of Deep Learning-Based Semantic Segmentation for Point Cloud","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"9680","DOI":"10.1109\/JSTARS.2021.3113047","article-title":"DAPnet: A Double Self-attention Convolutional Network for Point Cloud Semantic Labeling","volume":"14","author":"Chen","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.cag.2021.01.004","article-title":"PIG-Net: Inception based deep learning architecture for 3D point cloud segmentation","volume":"95","author":"Hegde","year":"2021","journal-title":"Comput. Graph."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"102633","DOI":"10.1016\/j.dsp.2019.102633","article-title":"Point-PlaneNet: Plane kernel based convolutional neural network for point clouds analysis","volume":"98","author":"Peyghambarzadeh","year":"2020","journal-title":"Digit. Signal Process."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hu, Q., Yang, B., Khalid, S., Xiao, W., Trigoni, N., and Markham, A. (2021, January 19\u201325). Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00494"},{"key":"ref_32","unstructured":"Sun, J., Zhang, Q., Kailkhura, B., Yu, Z., Xiao, C., and Mao, Z.M. (2022). Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions. arXiv."},{"key":"ref_33","unstructured":"Mao, J., Wang, X., and Li, H. (November, January 27). Interpolated convolutional networks for 3D point cloud understanding. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Hua, B.S., and Yeung, S.K. (November, January 27). ShellNet: Efficient Point Cloud Convolutional Neural Networks Using Concentric Shells Statistics. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00169"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Tatarchenko, M., Park, J., Koltun, V., and Zhou, Q.Y. (2018, January 18\u201322). Tangent Convolutions for Dense Prediction in 3D. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00409"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Wang, S., Suo, S., Ma, W.C., Pokrovsky, A., and Urtasun, R. (2018, January 18\u201322). Deep Parametric Continuous Convolutional Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00274"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Wu, W., Qi, Z., and Fuxin, L. (2019, January 15\u201320). PointCONV: Deep convolutional networks on 3D point clouds. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00985"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Qiu, Z., Li, Y., Wang, Y., Pan, Y., Yao, T., and Mei, T. (2022, January 23\u201327). SPE-Net: Boosting Point Cloud Analysis via Rotation Robustness Enhancement. Proceedings of the European Conference of Computer Vision (ECCV), Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-20062-5_34"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lin, H., Fan, B., Liu, Y., Yang, Y., Pan, Z., Shi, J., Pan, C., and Xie, H. (2021, January 10\u201315). PointSpherical: Deep Shape Context for Point Cloud Learning in Spherical Coordinates. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9412978"},{"key":"ref_40","first-page":"1","article-title":"Statistical Comparisons of Classifiers over Multiple Data Sets","volume":"7","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wilcox, R.R. (2003). Rank-based and nonparametric methods. Applying Contemporary Statistical Techniques, Academic Press.","DOI":"10.1016\/B978-012751541-0\/50036-5"},{"key":"ref_42","first-page":"335","article-title":"Benchmarking Deep Learning Models on Point Cloud Segmentation","volume":"Volume 339","author":"Zoumpekas","year":"2021","journal-title":"Artificial Intelligence Research and Development"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Chaton, T., Chaulet, N., Horache, S., and Landrieu, L. (2020, January 25\u201328). Torch-Points3D: A Modular Multi-Task Framework for Reproducible Deep Learning on 3D Point Clouds. Proceedings of the 2020 International Conference on 3D Vision (3DV), Fukuoka, Japan.","DOI":"10.1109\/3DV50981.2020.00029"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Tian, F., Jiang, Z., and Jiang, G. (2021). DNet: Dynamic Neighborhood Feature Learning in Point Cloud. Sensors, 21.","DOI":"10.3390\/s21072327"},{"key":"ref_45","unstructured":"Armeni, I., Sax, S., Zamir, A.R., and Savarese, S. (2017). Joint 2D-3D-Semantic Data for Indoor Scene Understanding. arXiv."},{"key":"ref_46","first-page":"102449","article-title":"MHIBS-Net: Multiscale hierarchical network for indoor building structure point clouds semantic segmentation","volume":"102","author":"Liang","year":"2021","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., and Markham, A. (2020, January 14\u201319). RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01112"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"3664","DOI":"10.1109\/TPAMI.2020.2983410","article-title":"Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds","volume":"43","author":"Lei","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Landrieu, L., and Boussaha, M. (2019, January 15\u201320). Point Cloud Oversegmentation with Graph-Structured Deep Metric Learning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00762"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"91","DOI":"10.5194\/isprs-annals-IV-1-W1-91-2017","article-title":"Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark","volume":"4","author":"Hackel","year":"2017","journal-title":"ISPRS Ann. Photogramm. Remote. Sens. Spat. Inf. Sci."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1016\/j.cag.2021.08.016","article-title":"ODFNet: Using orientation distribution functions to characterize 3D point clouds","volume":"102","author":"Sahin","year":"2020","journal-title":"Comput. Graph."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1228","DOI":"10.1007\/s11263-022-01601-z","article-title":"RIConv++: Effective Rotation Invariant Convolutions for 3D Point Clouds Deep Learning","volume":"130","author":"Zhang","year":"2022","journal-title":"Int. J. Comput. Vis."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L., and Tai, C.L. (2020, January 14\u201319). D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00639"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Fujiwara, K., and Hashimoto, T. (2020, January 14\u201319). Neural implicit embedding for point cloud analysis. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01175"},{"key":"ref_55","first-page":"4212","article-title":"Learning of 3D Graph Convolution Networks for Point Cloud Analysis","volume":"44","author":"Lin","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/23\/6049\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:29:26Z","timestamp":1760146166000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/23\/6049"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,29]]},"references-count":55,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["rs14236049"],"URL":"https:\/\/doi.org\/10.3390\/rs14236049","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,29]]}}}