{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:06:13Z","timestamp":1760144773350,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2024,5,8]],"date-time":"2024-05-08T00:00:00Z","timestamp":1715126400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Science Foundation of Shandong Province","award":["ZR2020MF045","2023RKL01004"],"award-info":[{"award-number":["ZR2020MF045","2023RKL01004"]}]},{"name":"Key Research and Development (R&amp;D) Plan of Shandong Province (Soft Science)","award":["ZR2020MF045","2023RKL01004"],"award-info":[{"award-number":["ZR2020MF045","2023RKL01004"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>In today\u2019s digital era, rumors spreading on social media threaten societal stability and individuals\u2019 daily lives, especially multimodal rumors. Hence, there is an urgent need for effective multimodal rumor detection methods. However, existing approaches often overlook the insufficient diversity of multimodal samples in feature space and hidden similarities and differences among multimodal samples. To address such challenges, we propose MVACLNet, a Multimodal Virtual Augmentation Contrastive Learning Network. In MVACLNet, we first design a Hierarchical Textual Feature Extraction (HTFE) module to extract comprehensive textual features from multiple perspectives. Then, we fuse the textual and visual features using a modified cross-attention mechanism, which operates from different perspectives at the feature value level, to obtain authentic multimodal feature representations. Following this, we devise a Virtual Augmentation Contrastive Learning (VACL) module as an auxiliary training module. It leverages ground-truth labels and extra-generated virtual multimodal feature representations to enhance contrastive learning, thus helping capture more crucial similarities and differences among multimodal samples. Meanwhile, it performs a Kullback\u2013Leibler (KL) divergence constraint between predicted probability distributions of the virtual multimodal feature representations and their corresponding virtual labels to help extract more content-invariant multimodal features. Finally, the authentic multimodal feature representations are input into a rumor classifier for detection. Experiments on two real-world datasets demonstrate the effectiveness and superiority of MVACLNet on multimodal rumor detection.<\/jats:p>","DOI":"10.3390\/a17050199","type":"journal-article","created":{"date-parts":[[2024,5,8]],"date-time":"2024-05-08T09:58:56Z","timestamp":1715162336000},"page":"199","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection"],"prefix":"10.3390","volume":"17","author":[{"given":"Xin","family":"Liu","sequence":"first","affiliation":[{"name":"Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China), No. 66, West Changjiang Road, Huangdao District, Qingdao 266580, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3152-1198","authenticated-orcid":false,"given":"Mingjiang","family":"Pang","sequence":"additional","affiliation":[{"name":"Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China), No. 66, West Changjiang Road, Huangdao District, Qingdao 266580, China"}]},{"given":"Qiang","family":"Li","sequence":"additional","affiliation":[{"name":"Scientific and Technological Innovation Center of ARI, Beijing 100020, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4026-1649","authenticated-orcid":false,"given":"Jiehan","family":"Zhou","sequence":"additional","affiliation":[{"name":"Information Technology and Electrical Engineering, University of Oulu, 90570 Oulu, Finland"}]},{"given":"Haiwen","family":"Wang","sequence":"additional","affiliation":[{"name":"Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China), No. 66, West Changjiang Road, Huangdao District, Qingdao 266580, China"}]},{"given":"Dawei","family":"Yang","sequence":"additional","affiliation":[{"name":"Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China), No. 66, West Changjiang Road, Huangdao District, Qingdao 266580, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,5,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"102025","DOI":"10.1016\/j.ipm.2019.03.004","article-title":"An overview of online fake news: Characterization, detection, and discussion","volume":"57","author":"Zhang","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1111\/hir.12320","article-title":"An exploration of how fake news is taking over social media and putting public health at risk","volume":"38","author":"Naeem","year":"2021","journal-title":"Health Inf. Libr. J."},{"key":"ref_3","unstructured":"Castillo, C., Mendoza, M., and Poblete, B. (April, January 28). Information credibility on twitter. Proceedings of the 20th International World Wide Web Conference, Hyderabad, India."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Resnick, P., and Mei, Q. (2015, January 18\u201322). Enquiring Minds: Early Detection of Rumors in Social Media from Enquiry Posts. Proceedings of the 24th International World Wide Web Conference, Florence, Italy.","DOI":"10.1145\/2736277.2741637"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Jin, Z., Cao, J., Zhang, Y., and Luo, J. (2016, January 12\u201317). News verification by exploiting conflicting social viewpoints in microblogs. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10382"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Shao, C., Ciampaglia, G.L., Flammini, A., and Menczer, F. (2016, January 11\u201315). Hoaxy: A platform for tracking online misinformation. Proceedings of the 25th International World Wide Web Conference, Montr\u00e9al, QC, Canada.","DOI":"10.1145\/2872518.2890098"},{"key":"ref_7","unstructured":"Ma, J., Gao, W., Mitra, P., Kwon, S., Jansen, B.J., Wong, K.F., and Cha, M. (2016, January 9\u201315). Detecting rumors from microblogs with recurrent neural networks. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yu, F., Liu, Q., Wu, S., Wang, L., and Tan, T. (2017, January 19\u201325). A Convolutional Approach for Misinformation Identification. Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/545"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ma, J., Gao, W., and Wong, K.F. (2018, January 23\u201327). Detect rumor and stance jointly by neural multi-task learning. Proceedings of the Companion Proceedings of the Web Conference 2018, Lyon, France.","DOI":"10.1145\/3184558.3188729"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Nan, Q., Cao, J., Zhu, Y., Wang, Y., and Li, J. (2021, January 1\u20135). MDFEND: Multi-domain fake news detection. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, Virtual Event.","DOI":"10.1145\/3459637.3482139"},{"key":"ref_11","first-page":"1242","article-title":"Category-controlled encoder-decoder for fake news detection","volume":"35","author":"Wu","year":"2023","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ma, J., Gao, W., and Wong, K.F. (2018, January 15\u201320). Rumor detection on Twitter with tree-structured recursive neural networks. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.","DOI":"10.18653\/v1\/P18-1184"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bian, T., Xiao, X., Xu, T., Zhao, P., Huang, W., Rong, Y., and Huang, J. (2020, January 7\u201312). Rumor detection on social media with bi-directional graph convolutional networks. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i01.5393"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wei, L., Hu, D., Zhou, W., Yue, Z., and Hu, S. (2021). Towards propagation uncertainty: Edge-enhanced bayesian graph convolutional networks for rumor detection. arXiv.","DOI":"10.18653\/v1\/2021.acl-long.297"},{"key":"ref_15","first-page":"1395","article-title":"A rumor detection approach based on multi-relational propagation tree","volume":"58","author":"Hu","year":"2021","journal-title":"J. Comput. Res. Dev."},{"key":"ref_16","unstructured":"Sun, M., Zhang, X., Zheng, J., and Ma, G. (March, January 22). DDGCN: Dual Dynamic Graph Convolutional Networks for Rumor Detection on Social Media. Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, Virtual Event."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1109\/TMM.2016.2617078","article-title":"Novel Visual and Statistical Image Features for Microblogs News Verification","volume":"19","author":"Jin","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_18","unstructured":"Alam, F., Cresci, S., Chakraborty, T., Silvestri, F., Dimitrov, D., Martino, G.D.S., Shaar, S., Firooz, H., and Nakov, P. (2021). A survey on multimodal disinformation detection. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Silva, A., Luo, L., Karunasekera, S., and Leckie, C. Embracing domain differences in fake news: Cross-domain fake news detection using multi-modal data. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, Virtual Event, 2\u20139 February 2021.","DOI":"10.1609\/aaai.v35i1.16134"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., Su, L., and Gao, J. (2018, January 19\u201323). EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3219903"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Khattar, D., Goud, J.S., Gupta, M., and Varma, V. (2019, January 13\u201317). MVAE: Multimodal Variational Autoencoder for Fake News Detection. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.","DOI":"10.1145\/3308558.3313552"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhou, X., Wu, J., and Zafarani, R. (2020, January 11\u201314). SAFE: Similarity-Aware Multi-modal Fake News Detection. Proceedings of the 24th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, Singapore.","DOI":"10.1007\/978-3-030-47436-2_27"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Singhal, S., Shah, R.R., Chakraborty, T., Kumaraguru, P., and Satoh, S.I. (2019, January 11\u201313). Spotfake: A multi-modal framework for fake news detection. Proceedings of the 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), Singapore.","DOI":"10.1109\/BigMM.2019.00-44"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Singhal, S., Kabra, A., Sharma, M., Shah, R.R., Chakraborty, T., and Kumaraguru, P. (2020, January 7\u201312). Spotfake+: A multimodal framework for fake news detection via transfer learning (student abstract). Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i10.7230"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jin, Z., Cao, J., Guo, H., Zhang, Y., and Luo, J. (2017, January 23\u201327). Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs. Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, CA, USA.","DOI":"10.1145\/3123266.3123454"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Qian, S., Wang, J., Hu, J., Fang, Q., and Xu, C. (2021, January 11\u201315). Hierarchical multi-modal contextual attention network for fake news detection. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event.","DOI":"10.1145\/3404835.3462871"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wu, Y., Zhan, P., Zhang, Y., Wang, L., and Xu, Z. (2021, January 1\u20136). Multimodal fusion with co-attention networks for fake news detection. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Bangkok, Thailand.","DOI":"10.18653\/v1\/2021.findings-acl.226"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"102610","DOI":"10.1016\/j.ipm.2021.102610","article-title":"Detecting fake news by exploring the consistency of multimodal data","volume":"58","author":"Xue","year":"2021","journal-title":"Inf. Process. Manag."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Chen, Y., Li, D., Zhang, P., Sui, J., Lv, Q., Tun, L., and Shang, L. (2022, January 25\u201329). Cross-modal Ambiguity Learning for Multimodal Fake News Detection. Proceedings of the ACM Web Conference 2022, Virtual Event.","DOI":"10.1145\/3485447.3511968"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"103822","DOI":"10.1016\/j.csi.2023.103822","article-title":"MRAN: Multimodal relationship-aware attention network for fake news detection","volume":"89","author":"Yang","year":"2024","journal-title":"Comput. Stand. Interfaces"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Qi, P., Cao, J., Yang, T., Guo, J., and Li, J. (2019, January 8\u201311). Exploiting Multi-domain Visual Information for Fake News Detection. Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), Beijing, China.","DOI":"10.1109\/ICDM.2019.00062"},{"key":"ref_32","first-page":"898","article-title":"Contrastive Learning for Image Captioning","volume":"30","author":"Dai","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cai, H., Chen, H., Song, Y., Ding, Z., Bao, Y., Yan, W., and Zhao, X. (2020). Group-wise Contrastive Learning for Neural Dialogue Generation. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.70"},{"key":"ref_34","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13\u201318). A Simple Framework for Contrastive Learning of Visual Representations. Proceedings of the 37th International Conference on Machine Learning, Virtual Event."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wu, H., Ma, T., Wu, L., Manyumwa, T., and Ji, S. (2020). Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.294"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Xu, Y., Yu, F., Liu, Q., Wu, S., and Wang, L. (2021, January 19\u201323). Graph Contrastive Learning with Adaptive Augmentation. Proceedings of the Web Conference 2021, Ljubljana, Slovenia.","DOI":"10.1145\/3442381.3449802"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Sun, T., Qian, Z., Dong, S., Li, P., and Zhu, Q. (2022, January 25\u201329). Rumor Detection on Social Media with Graph Adversarial Contrastive Learning. Proceedings of the ACM Web Conference 2022, Virtual Event.","DOI":"10.1145\/3485447.3511999"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Chen, J., Yang, Z., and Yang, D. (2020). MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.194"},{"key":"ref_39","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. arXiv.","DOI":"10.3115\/v1\/D14-1181"},{"key":"ref_41","unstructured":"Yao, L., Mao, C., and Luo, Y. (February, January 27). Graph Convolutional Networks for Text Classification. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_42","unstructured":"Kipf, T.N., and Welling, M. (2016). Semi-Supervised Classification with Graph Convolutional Networks. arXiv."},{"key":"ref_43","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_44","first-page":"12","article-title":"ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks","volume":"32","author":"Lu","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_45","first-page":"6000","article-title":"Attention is All you Need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/s13735-017-0143-x","article-title":"Detection and visualization of misleading content on Twitter","volume":"7","author":"Boididou","year":"2018","journal-title":"Int. J. Multimed. Inf. Retr."},{"key":"ref_47","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4\u20139). Automatic differentiation in PyTorch. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_48","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_50","unstructured":"Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. arXiv."},{"key":"ref_51","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_52","first-page":"5754","article-title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","volume":"32","author":"Yang","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_53","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/5\/199\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:41:35Z","timestamp":1760107295000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/5\/199"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,8]]},"references-count":53,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["a17050199"],"URL":"https:\/\/doi.org\/10.3390\/a17050199","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2024,5,8]]}}}