{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T16:53:55Z","timestamp":1790960035464,"version":"4.1.0"},"publisher-location":"Cham","reference-count":40,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031783463","type":"print"},{"value":"9783031783470","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>3D scene segmentation is a crucial task in Computer Vision, with applications in autonomous driving, augmented reality, and robotics. Traditional methods often struggle to provide consistent and accurate segmentation across different viewpoints.\u00a0To address this, we look at the growing field of novel view synthesis. Methods like NeRF and 3DGS take a set of images and implicitly\u00a0learn a multi-view consistent representation of the geometry of the scene; the same strategy can be extended to learn a 3D segmentation of\u00a0the scene that is consistent with the 2D segmentation of an initial training set of input images.<\/jats:p><jats:p>We introduce <jats:italic>Contrastive Gaussian Clustering<\/jats:italic>, a\u00a0novel approach for novel segmentation view synthesis and 3D\u00a0scene segmentation. We extend 3D Gaussian Splatting to include a learnable 3D feature field, which allows us to cluster the 3D Gaussians\u00a0into objects. Using a combination of contrastive learning and spatial regularization, our model can be trained on inconsistent\u00a02D segmentation labels, and still learn to generate multi-view consistent masks. Moreover, the resulting model is extremely accurate, improving the IoU accuracy of the predicted masks\u00a0by <jats:inline-formula><jats:alternatives><jats:tex-math>$$+8\\%$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                <mml:mrow>\n                  <mml:mo>+<\/mml:mo>\n                  <mml:mn>8<\/mml:mn>\n                  <mml:mo>%<\/mml:mo>\n                <\/mml:mrow>\n              <\/mml:math><\/jats:alternatives><\/jats:inline-formula> over the state of the art.<\/jats:p><jats:p>Code and trained models are available\u00a0at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/MyrnaCCS\/contrastive-gaussian-clustering\">https:\/\/github.com\/MyrnaCCS\/contrastive-gaussian-clustering<\/jats:ext-link>.<\/jats:p>","DOI":"10.1007\/978-3-031-78347-0_8","type":"book-chapter","created":{"date-parts":[[2024,12,1]],"date-time":"2024-12-01T14:54:18Z","timestamp":1733064858000},"page":"114-130","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Contrastive Gaussian Clustering for\u00a0Weakly Supervised 3D Scene Segmentation"],"prefix":"10.1007","author":[{"given":"Myrna","family":"Castillo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahtab","family":"Dahaghin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matteo","family":"Toso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessio","family":"Del Bue","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,12,2]]},"reference":[{"key":"8_CR1","doi-asserted-by":"crossref","unstructured":"Barron, J.T., et\u00a0al.: Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: CVPR (2022)","DOI":"10.1109\/CVPR52688.2022.00539"},{"key":"8_CR2","doi-asserted-by":"crossref","unstructured":"Behley, J., et al.: SemanticKITTI: a dataset for semantic scene understanding of lidar sequences. ICCV (2019)","DOI":"10.1109\/ICCV.2019.00939"},{"key":"8_CR3","unstructured":"Bhalgat, Y., Laina, I., Henriques, J.F., Zisserman, A., Vedaldi, A.: Contrastive lift: 3D object instance segmentation by slow-fast contrastive fusion. In: NeurIPS (2023)"},{"key":"8_CR4","doi-asserted-by":"crossref","unstructured":"Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: ICCV (2021)","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"8_CR5","unstructured":"Cen, J., et al.: Segment any 3D Gaussians. arXiv preprint arXiv:2312.00860 (2023)"},{"key":"8_CR6","doi-asserted-by":"crossref","unstructured":"Chang, A., et al.: Matterport3D: learning from RGB-D data in indoor environments. 3DV (2017)","DOI":"10.1109\/3DV.2017.00081"},{"key":"8_CR7","doi-asserted-by":"crossref","unstructured":"Chen, D.Z., Chang, A.X., Nie\u00dfner, M.: Scanrefer: 3D object localization in RGB-D scans using natural language. In: ECCV (2020)","DOI":"10.1007\/978-3-030-58565-5_13"},{"key":"8_CR8","unstructured":"Chen, G., Wang, W.: A Survey on 3D Gaussian Splatting (2024)"},{"key":"8_CR9","doi-asserted-by":"crossref","unstructured":"Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nie\u00dfner, M.: ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In: CVPR (2017)","DOI":"10.1109\/CVPR.2017.261"},{"key":"8_CR10","doi-asserted-by":"crossref","unstructured":"Fang, J., Wang, J., Zhang, X., Xie, L., Tian, Q.: GaussianEditor: editing 3D Gaussians delicately with text instructions. arXiv preprint arXiv:2311.16037 (2023)","DOI":"10.1109\/CVPR52733.2024.01975"},{"key":"8_CR11","doi-asserted-by":"crossref","unstructured":"Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Garcia-Rodriguez, J.: A review on deep learning techniques applied to semantic segmentation (2017)","DOI":"10.1016\/j.asoc.2018.05.018"},{"key":"8_CR12","doi-asserted-by":"crossref","unstructured":"Genova, K., et al.: Learning 3D semantic segmentation with only 2D image supervision. 3DV (2021)","DOI":"10.1109\/3DV53792.2021.00046"},{"key":"8_CR13","doi-asserted-by":"crossref","unstructured":"Hou, J., Dai, A., Nie\u00dfner, M.: 3D-SIS: 3D semantic instance segmentation of RGB-D scans. In: CVPR (2019)","DOI":"10.1109\/CVPR.2019.00455"},{"key":"8_CR14","doi-asserted-by":"crossref","unstructured":"Hu, Q., et al.: Randla-net: efficient semantic segmentation of large-scale point clouds. In: CVPR (2020)","DOI":"10.1109\/CVPR42600.2020.01112"},{"key":"8_CR15","doi-asserted-by":"crossref","unstructured":"Hua, B.S., Pham, Q.H., Nguyen, D.T., Tran, M.K., Yu, L.F., Yeung, S.K.: SceneNN: a scene meshes dataset with annotations. In: 3DV (2016)","DOI":"10.1109\/3DV.2016.18"},{"key":"8_CR16","doi-asserted-by":"crossref","unstructured":"Kerbl, B., Kopanas, G., Leimk\u00fchler, T., Drettakis, G.: 3D Gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. (2023)","DOI":"10.1145\/3592433"},{"key":"8_CR17","doi-asserted-by":"crossref","unstructured":"Kerr, J., Kim, C.M., Goldberg, K., Kanazawa, A., Tancik, M.: LERF: language embedded radiance fields. In: ICCV (2023)","DOI":"10.1109\/ICCV51070.2023.01807"},{"key":"8_CR18","doi-asserted-by":"crossref","unstructured":"Kirillov, A., et al.: Segment anything. arXiv:2304.02643 (2023)","DOI":"10.1109\/ICCV51070.2023.00371"},{"key":"8_CR19","doi-asserted-by":"crossref","unstructured":"Kundu, A., et\u00a0al.: Panoptic neural fields: a semantic object-aware neural scene representation. In: CVPR (2022)","DOI":"10.1109\/CVPR52688.2022.01253"},{"key":"8_CR20","unstructured":"Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: ICLR (2022)"},{"key":"8_CR21","doi-asserted-by":"crossref","unstructured":"Liao, Y., Xie, J., Geiger, A.: KITTI-360: a novel dataset and benchmarks for urban scene understanding in 2D and 3D. TPAMI (2023)","DOI":"10.1109\/TPAMI.2022.3179507"},{"key":"8_CR22","unstructured":"Liu, K., et al.: Weakly supervised 3D open-vocabulary segmentation. In: NeurIPS (2023)"},{"key":"8_CR23","doi-asserted-by":"crossref","unstructured":"Liu, S., et\u00a0al.: Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)","DOI":"10.1007\/978-3-031-72970-6_3"},{"key":"8_CR24","doi-asserted-by":"crossref","unstructured":"Liu, Y., Fan, Q., Zhang, S., Dong, H., Funkhouser, T.A., Yi, L.: Contrastive multimodal fusion with tupleinfonce. ICCV (2021)","DOI":"10.1109\/ICCV48922.2021.00079"},{"key":"8_CR25","doi-asserted-by":"crossref","unstructured":"Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"8_CR26","doi-asserted-by":"crossref","unstructured":"M\u00fcller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. (2022)","DOI":"10.1145\/3528223.3530127"},{"key":"8_CR27","doi-asserted-by":"crossref","unstructured":"Naseer, M., Khan, S., Porikli, F.: Indoor scene understanding in 2.5\/3D for autonomous agents: a survey. IEEE Access (2019)","DOI":"10.1109\/ACCESS.2018.2886133"},{"key":"8_CR28","doi-asserted-by":"crossref","unstructured":"Peng, S., Genova, K., Jiang, C.M., Tagliasacchi, A., Pollefeys, M., Funkhouser, T.: OpenScene: 3D scene understanding with open vocabularies (2023)","DOI":"10.1109\/CVPR52729.2023.00085"},{"key":"8_CR29","unstructured":"Qi, C.R., et\u00a0al.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: NeurIPS (2017)"},{"key":"8_CR30","unstructured":"Qi, C., Su, H., Mo, K., Guibas, L.: Pointnet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652\u2013660 (2017)"},{"key":"8_CR31","doi-asserted-by":"crossref","unstructured":"Qin, M., Li, W., Zhou, J., Wang, H., Pfister, H.: LangSplat: 3D language Gaussian splatting (2023)","DOI":"10.1109\/CVPR52733.2024.01895"},{"key":"8_CR32","unstructured":"Radford, A., et\u00a0al.: Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning, pp. 8748\u20138763. PMLR (2021)"},{"key":"8_CR33","doi-asserted-by":"crossref","unstructured":"Sautier, C., Puy, G., Gidaris, S., Boulch, A., Bursuc, A., Marlet, R.: Image-to-lidar self-supervised distillation for autonomous driving data. In: CVPR (2022)","DOI":"10.1109\/CVPR52688.2022.00966"},{"key":"8_CR34","unstructured":"Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing NeRF for editing via feature field distillation. In: NeuIPS (2022)"},{"key":"8_CR35","doi-asserted-by":"crossref","unstructured":"Wu, G., et al.: 4D Gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023)","DOI":"10.1109\/CVPR52733.2024.01920"},{"key":"8_CR36","unstructured":"Wu, Z., et al.: 3D ShapeNets: a deep representation for volumetric shapes. In: CVPR (2015)"},{"key":"8_CR37","doi-asserted-by":"crossref","unstructured":"Ye, M., Danelljan, M., Yu, F., Ke, L.: Gaussian grouping: segment and edit anything in 3D scenes. arXiv preprint arXiv:2312.00732 (2023)","DOI":"10.1007\/978-3-031-73397-0_10"},{"key":"8_CR38","doi-asserted-by":"crossref","unstructured":"Ying, H., et al.: Omniseg3D: Omniversal 3D segmentation via hierarchical contrastive learning (2023)","DOI":"10.1109\/CVPR52733.2024.01948"},{"key":"8_CR39","doi-asserted-by":"crossref","unstructured":"Yu, A., Fridovich-Keil, S., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: radiance fields without neural networks (2021)","DOI":"10.1109\/CVPR52688.2022.00542"},{"key":"8_CR40","doi-asserted-by":"crossref","unstructured":"Zhi, S., Laidlow, T., Leutenegger, S., Davison, A.J.: In-place scene labelling and understanding with implicit scene representation. In: ICCV (2021)","DOI":"10.1109\/ICCV48922.2021.01554"}],"container-title":["Lecture Notes in Computer Science","Pattern Recognition"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-78347-0_8","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,1]],"date-time":"2024-12-01T15:03:32Z","timestamp":1733065412000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-78347-0_8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,2]]},"ISBN":["9783031783463","9783031783470"],"references-count":40,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-78347-0_8","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,2]]},"assertion":[{"value":"2 December 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ICPR","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Pattern Recognition","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Kolkata","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"India","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1 December 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"5 December 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"icpr2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/icpr2024.org\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}