{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,22]],"date-time":"2025-11-22T07:03:13Z","timestamp":1763794993841,"version":"3.45.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"12","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["6207071897"],"award-info":[{"award-number":["6207071897"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>Self-supervised learning has shown remarkable effectiveness in 3D point cloud understanding. Existing masked autoencoders or contrastive learning paradigms can acquire robust feature representations from unlabeled data. Specifically, masked autoencoders extract features of local patches and directly map them to latent global vectors, suffering from insufficient semantic extraction and latent interaction. Contrastive paradigms capture global correspondence via restricted constraints and are limited by the absence of local detail modeling. This prompts us to integrate the synergistic local and global advantages of two effective components and extend them further for multimodal dependencies. In this article, we propose a unified correlation-guided masked autoencoder with multimodal contrastive interaction (CorMAC) learning for self-supervised point cloud analysis. We first design the spherical adaptive embedding backbone to learn local underlying semantics and improve the masked mechanism for patch autoencoding and reconstruction. Then, we expand multimodal contrastive correspondence and constraints to leverage the potential alignments across point cloud and auxiliary image modalities. In addition, we devise adaptable loss functions to jointly optimize masked recover and contrast errors, aiming to enhance latent feature learning. Extensive experiments show that our method achieves superior performance than other self-supervised ones on various datasets and exhibits better generalization capability across diverse downstream tasks.<\/jats:p>","DOI":"10.1145\/3770579","type":"journal-article","created":{"date-parts":[[2025,10,3]],"date-time":"2025-10-03T16:09:24Z","timestamp":1759507764000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Correlation-guided Masked Autoencoder with Multimodal Contrastive Interaction on Point Clouds"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4195-3986","authenticated-orcid":false,"given":"Peng","family":"Ren","sequence":"first","affiliation":[{"name":"Tongji University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-6069-6912","authenticated-orcid":false,"given":"Xiaoheng","family":"Li","sequence":"additional","affiliation":[{"name":"Tongji University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2042-9237","authenticated-orcid":false,"given":"Yunfeng","family":"Bai","sequence":"additional","affiliation":[{"name":"Tongji University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4452-1396","authenticated-orcid":false,"given":"Jinyuan","family":"Jia","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,11,21]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00967"},{"key":"e_1_3_1_3_2","unstructured":"Hangbo Bao Li Dong Songhao Piao and Furu Wei. 2021. BEiT: BERT pre-training of image transformers. arXiv:2106.08254. Retrieved from https:\/\/arxiv.org\/abs\/2106.08254"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00951"},{"key":"e_1_3_1_5_2","unstructured":"Angel X. Chang Thomas Funkhouser Leonidas Guibas Pat Hanrahan Qixing Huang Zimo Li Silvio Savarese Manolis Savva Shuran Song Hao Su et al. 2015. ShapeNet: An information-rich 3D model repository. arXiv:1512.03012. Retrieved from https:\/\/arxiv.org\/abs\/1512.03012"},{"key":"e_1_3_1_6_2","first-page":"1597","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Chen Ting","year":"2020","unstructured":"Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. PMLR, 1597\u20131607."},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2021.3072214"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00912"},{"key":"e_1_3_1_9_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Dong Runpei","year":"2023","unstructured":"Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. 2023. Autoencoders as cross-modal teachers: Can pretrained 2D image transformers help 3D representation learning? In Proceedings of the 11th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=8Oun8ZUVe8N"},{"key":"e_1_3_1_10_2","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929. Retrieved from https:\/\/arxiv.org\/abs\/2010.11929"},{"key":"e_1_3_1_11_2","volume-title":"Proceedings of the International Conference on Learning Representations. Retrieved from","author":"Dosovitskiy Alexey","year":"2021","unstructured":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/pdf?id=YicbFdNTTy"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475458"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.264"},{"key":"e_1_3_1_14_2","unstructured":"P. Gao T. Ma H. Li Z. Lin J. Dai and Y. Qiao. 2022. ConvMAE: Masked convolution meets masked autoencoders. arXiv:2205.03892. Retrieved from https:\/\/arxiv.org\/abs\/2205.03892"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2022.3151665"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s41095-021-0229-5"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01553"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00831"},{"key":"e_1_3_1_20_2","first-page":"820","article-title":"PointCNN: Convolution on X-transformed points","volume":"31","author":"Li Yangyan","year":"2018","unstructured":"Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018. PointCNN: Convolution on X-transformed points. In Advances in Neural Information Processing Systems, Vol. 31, 820\u2013830.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.170"},{"issue":"8","key":"e_1_3_1_22_2","first-page":"4212","article-title":"Learning of 3D graph convolution networks for point cloud analysis","volume":"44","author":"Lin Zhi-Hao","year":"2021","unstructured":"Zhi-Hao Lin, Sheng-Yu Huang, and Yu-Chiang Frank Wang. 2021. Learning of 3D graph convolution networks for point cloud analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 8 (2021), 4212\u20134224.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/3DV57658.2022.00017"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20086-1_38"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639470"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00910"},{"key":"e_1_3_1_27_2","unstructured":"Yueh-Cheng Liu Yu-Kai Huang Hung-Yueh Chiang Hung-Ting Su Zhe-Yu Liu Chin-Tang Chen Ching-Yu Tseng and Winston H. Hsu. 2021. Learning from 2D: Contrastive pixel-to-point knowledge transfer for 3D pretraining. arXiv:2104.04687. Retrieved from https:\/\/arxiv.org\/abs\/2104.04687"},{"key":"e_1_3_1_28_2","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101. Retrieved from https:\/\/arxiv.org\/abs\/1711.05101."},{"key":"e_1_3_1_29_2","unstructured":"Xu Ma Can Qin Haoxuan You Haoxi Ran and Yun Fu. 2022. Rethinking network design and local geometry in point cloud: A simple residual MLP framework. arXiv:2202.07123. Retrieved from https:\/\/arxiv.org\/abs\/2202.07123"},{"key":"e_1_3_1_30_2","first-page":"412","volume-title":"Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing","author":"Madasu Avinash","year":"2019","unstructured":"Avinash Madasu and Vijjini Anvesh Rao. 2019. Effectiveness of self normalizing neural networks for text classification. In Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing. Springer, 412\u2013423."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3617833"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00674"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3060413"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20086-1_35"},{"key":"e_1_3_1_35_2","first-page":"652","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Qi Charles R.","year":"2017","unstructured":"Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 652\u2013660."},{"key":"e_1_3_1_36_2","first-page":"5099","article-title":"PointNet++: deep hierarchical feature learning on point sets in a metric space","volume":"30","author":"Qi Charles Ruizhongtai","year":"2017","unstructured":"Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. 2017. PointNet++: deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, Vol. 30, 5099\u20135108.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_37_2","first-page":"23192","article-title":"PointNeXt: Revisiting PointNet++ with improved training and scaling strategies","volume":"35","author":"Qian Guocheng","year":"2022","unstructured":"Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. 2022. PointNeXt: Revisiting PointNet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems 35, (2022), 23192\u201323204.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2021.3074240"},{"key":"e_1_3_1_39_2","first-page":"12962","article-title":"Self-supervised deep learning on point clouds by reconstructing space","volume":"32","author":"Sauder Jonathan","year":"2019","unstructured":"Jonathan Sauder and Bjarne Sievers. 2019. Self-supervised deep learning on point clouds by reconstructing space. In Advances in Neural Information Processing Systems, Vol. 32, 12962\u201312972.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_40_2","first-page":"7212","article-title":"Self-supervised few-shot learning on point clouds","volume":"33","author":"Sharma Charu","year":"2020","unstructured":"Charu Sharma and Manohar Kaul. 2020. Self-supervised few-shot learning on point clouds. In Advances in Neural Information Processing Systems, Vol. 33, 7212\u20137221.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00651"},{"key":"e_1_3_1_42_2","first-page":"10347","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Touvron Hugo","year":"2021","unstructured":"Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv\u00e9 J\u00e9gou. 2021. Training data-efficient image transformers & distillation through attention. In Proceedings of the International Conference on Machine Learning. PMLR, 10347\u201310357."},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00167"},{"key":"e_1_3_1_44_2","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, 5998\u20136008.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00964"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS51168.2021.9636655"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00304"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3326362"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3056238"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2023.3284591"},{"key":"e_1_3_1_51_2","first-page":"1912 1920","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Wu Zhirong","year":"2015","unstructured":"Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1912\u20131920."},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58580-8_34"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00943"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01237-3_6"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980238"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01270-0_35"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01871"},{"key":"e_1_3_1_58_2","first-page":"12310","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zbontar Jure","year":"2021","unstructured":"Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St\u00e9phane Deny. 2021. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the International Conference on Machine Learning. PMLR, 12310\u201312320."},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","unstructured":"Renrui Zhang Liuhui Wang Ziyu Guo Yali Wang Peng Gao Hongsheng Li and Jianbo Shi. 2023. Parameter is not all you need: Starting from non-parametric networks for 3D point cloud analysis. arXiv:2303.08134. Retrieved from https:\/\/arxiv.org\/abs\/2303.08134.","DOI":"10.1109\/CVPR52729.2023.00517"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01009"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3538648"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3770579","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,22]],"date-time":"2025-11-22T06:58:54Z","timestamp":1763794734000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3770579"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,21]]},"references-count":60,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3770579"],"URL":"https:\/\/doi.org\/10.1145\/3770579","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"type":"print","value":"1551-6857"},{"type":"electronic","value":"1551-6865"}],"subject":[],"published":{"date-parts":[[2025,11,21]]},"assertion":[{"value":"2024-09-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}