{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T21:35:42Z","timestamp":1792100142117,"version":"4.4.0"},"reference-count":44,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2022,11,23]],"date-time":"2022-11-23T00:00:00Z","timestamp":1669161600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62166042"],"award-info":[{"award-number":["62166042"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U2003207"],"award-info":[{"award-number":["U2003207"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Natural Science Foundation of Xinjiang, China","award":["2021D01C076"],"award-info":[{"award-number":["2021D01C076"]}]},{"name":"Strengthening Plan of National Defense Science and Technology Foundation of China","award":["2021-JCJQ-JJ-0059"],"award-info":[{"award-number":["2021-JCJQ-JJ-0059"]}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>False information detection can detect false information in social media and reduce its negative impact on society. With the development of multimedia, the multimodal content contained in false information is increasing, so it is important to use multimodal features to detect false information. This paper mainly uses information from two modalities, text and image. The features extracted by the backbone network are not further processed in the previous work, and the problems of noise and information loss in the process of fusing multimodal features are ignored. This paper proposes a false information detection method based on Text-CNN and SE modules. We use Text-CNN to process the text and image features extracted by BERT and Swin-transformer to enhance the quality of the features. In addition, we use the modified SE module to fuse text and image features and reduce the noise in the fusion process. Meanwhile, we draw on the idea of residual networks to reduce information loss in the fusion process by concatenating the original features with the fused features. Our model improves accuracy by 6.5% and 2.0% on the Weibo dataset and Twitter dataset compared to the attention based multimodal factorized bilinear pooling. The comparative experimental results show that the proposed model can improve the accuracy of false information detection. The results of ablation experiments further demonstrate the effectiveness of each module in our model.<\/jats:p>","DOI":"10.1371\/journal.pone.0277463","type":"journal-article","created":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T14:31:29Z","timestamp":1669300289000},"page":"e0277463","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":16,"title":["Multimodal false information detection method based on Text-CNN and SE module"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6481-0692","authenticated-orcid":true,"given":"Yi","family":"Liang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Turdi","family":"Tohti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Askar","family":"Hamdulla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,11,23]]},"reference":[{"key":"pone.0277463.ref001","doi-asserted-by":"crossref","unstructured":"Gupta, Manish and Zhao, Peixiang and Han, Jiawei. Evaluating event credibility on twitter. Proceedings of the 2012 SIAM international conference on data mining. Society for Industrial and Applied Mathematics, 2012: 153-164.","DOI":"10.1137\/1.9781611972825.14"},{"issue":"3","key":"pone.0277463.ref002","first-page":"7","article-title":"Verifying multimedia use at mediaeval","volume":"3","author":"C Boididou","year":"2015","journal-title":"MediaEval"},{"key":"pone.0277463.ref003","doi-asserted-by":"crossref","unstructured":"Rashkin H, Choi E, Jang J Y, et al. Truth of varying shades: Analyzing language in fake news and political fact-checking Proceedings of the 2017 conference on empirical methods in natural language processing. 2017: 2931-2937.","DOI":"10.18653\/v1\/D17-1317"},{"key":"pone.0277463.ref004","doi-asserted-by":"crossref","unstructured":"Popat K, Mukherjee S, Str\u00f6tgen J, et al. Credibility assessment of textual claims on the web. Proceedings of the 25th ACM international on conference on information and knowledge management. 2016: 2173-2178.","DOI":"10.1145\/2983323.2983661"},{"key":"pone.0277463.ref005","unstructured":"P\u00e9rez-Rosas V, Kleinberg B, Lefevre A, et al Automatic detection of fake news. arXiv preprint arXiv:1708.07104, 2017."},{"key":"pone.0277463.ref006","doi-asserted-by":"crossref","unstructured":"Alonso-Bartolome S, Segura-Bedmar I. Multimodal Fake News Detection. arXiv preprint arXiv:2112.04831, 2021.","DOI":"10.3390\/info13060284"},{"issue":"10","key":"pone.0277463.ref007","doi-asserted-by":"crossref","first-page":"13799","DOI":"10.1007\/s11042-022-12290-8","article-title":"An effective strategy for multi-modal fake news detection","volume":"81","author":"X Peng","year":"2022","journal-title":"Multimedia Tools and Applications"},{"key":"pone.0277463.ref008","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1016\/j.patrec.2022.01.007","article-title":"Effective fake news video detection using domain knowledge and multimodal data fusion on youtube","volume":"154","author":"H Choi","year":"2022","journal-title":"Pattern Recognition Letters"},{"key":"pone.0277463.ref009","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/978-3-642-24797-2_4","article-title":"Long short-term memory","author":"A Graves","year":"2012","journal-title":"Supervised sequence labelling with recurrent neural networks"},{"key":"pone.0277463.ref010","article-title":"Attention is all you need","volume":"30","author":"A Vaswani","year":"2017","journal-title":"Advances in neural information processing systems"},{"issue":"6","key":"pone.0277463.ref011","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"60","author":"A Krizhevsky","year":"2017","journal-title":"Communications of the ACM"},{"key":"pone.0277463.ref012","volume-title":"Convolutional neural network for sentence classification","author":"Y Chen","year":"2015"},{"key":"pone.0277463.ref013","doi-asserted-by":"crossref","unstructured":"Hu J, Shen L, Sun G. Squeeze-and-excitation networks. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"pone.0277463.ref014","unstructured":"Lee J D M C K, Toutanova K. Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018."},{"key":"pone.0277463.ref015","doi-asserted-by":"crossref","unstructured":"Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE\/CVF International Conference on Computer Vision. 2021: 10012-10022.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"pone.0277463.ref016","doi-asserted-by":"crossref","unstructured":"Jin Z, Cao J, Guo H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. Proceedings of the 25th ACM international conference on Multimedia. 2017: 795-816.","DOI":"10.1145\/3123266.3123454"},{"key":"pone.0277463.ref017","unstructured":"Qazvinian V, Rosengren E, Radev D, et al. Rumor has it: Identifying misinformation in microblogs. Proceedings of the 2011 conference on empirical methods in natural language processing. 2011: 1589-1599."},{"issue":"2","key":"pone.0277463.ref018","first-page":"419","article-title":"Multi-modal deep fusion for false information detection","volume":"42","author":"J MENG","year":"2022","journal-title":"Journal of Computer Applications"},{"issue":"11","key":"pone.0277463.ref019","article-title":"Rumor detection based on convolutional neural network","volume":"37","author":"Z LIU","year":"2017","journal-title":"Journal of Computer Applications"},{"key":"pone.0277463.ref020","author":"J Ma","year":"2016","journal-title":"Detecting rumors from microblogs with recurrent neural networks"},{"issue":"1","key":"pone.0277463.ref021","doi-asserted-by":"crossref","first-page":"100007","DOI":"10.1016\/j.jjimei.2020.100007","article-title":"Fake news detection: A hybrid CNN-RNN based deep learning approach","volume":"1","author":"J A Nasir","year":"2021","journal-title":"International Journal of Information Management Data Insights"},{"key":"pone.0277463.ref022","doi-asserted-by":"crossref","unstructured":"Singhal S, Shah R R, Chakraborty T, et al. Spotfake: A multi-modal framework for fake news detection. 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, 2019: 39-47.","DOI":"10.1109\/BigMM.2019.00-44"},{"key":"pone.0277463.ref023","doi-asserted-by":"crossref","first-page":"115412","DOI":"10.1016\/j.eswa.2021.115412","article-title":"Amfb: attention based multimodal factorized bilinear pooling for multimodal fake news detection","volume":"184","author":"R Kumari","year":"2021","journal-title":"Expert Systems with Applications"},{"issue":"1","key":"pone.0277463.ref024","doi-asserted-by":"crossref","first-page":"102437","DOI":"10.1016\/j.ipm.2020.102437","article-title":"A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks","volume":"58","author":"C Song","year":"2021","journal-title":"Information Processing Management"},{"key":"pone.0277463.ref025","unstructured":"Dhawan M, Sharma S, Kadam A, et al. GAME-ON: Graph Attention Network based Multimodal Fusion for Fake News Detection. arXiv preprint arXiv:2202.12478, 2022."},{"key":"pone.0277463.ref026","first-page":"2560","article-title":"Multimodal fusion with co-attention networks for fake news detection","author":"Y Wu","year":"2021","journal-title":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021"},{"key":"pone.0277463.ref027","unstructured":"Xu P, Zhu X, Clifton D A. Multimodal Learning with Transformers: A Survey. arXiv preprint arXiv:2206.06488, 2022."},{"key":"pone.0277463.ref028","doi-asserted-by":"crossref","unstructured":"Gavrilyuk K, Sanford R, Javan M, et al. Actor-transformers for group activity recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 2020: 839-848.","DOI":"10.1109\/CVPR42600.2020.00092"},{"key":"pone.0277463.ref029","unstructured":"Xu P, Zhu X. Deepchange: A large long-term person re-identification benchmark with clothes change. arXiv e-prints, 2021: arXiv: 2105.14685."},{"key":"pone.0277463.ref030","doi-asserted-by":"crossref","unstructured":"Sun C, Myers A, Vondrick C, et al. Videobert: A joint model for video and language representation learning. Proceedings of the IEEE\/CVF International Conference on Computer Vision. 2019: 7464-7473.","DOI":"10.1109\/ICCV.2019.00756"},{"key":"pone.0277463.ref031","unstructured":"Guo D, Ren S, Lu S, et al. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366, 2020."},{"key":"pone.0277463.ref032","unstructured":"Shi B, Hsu W N, Lakhotia K, et al. Learning audio-visual speech representation by masked multimodal cluster prediction. arXiv preprint arXiv:2201.02184, 2022."},{"key":"pone.0277463.ref033","unstructured":"Zheng R, Chen J, Ma M, et al. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. International Conference on Machine Learning. PMLR, 2021: 12736-12746."},{"key":"pone.0277463.ref034","doi-asserted-by":"crossref","unstructured":"Li R, Yang S, Ross D A, et al. Ai choreographer: Music conditioned 3d dance generation with aist++. Proceedings of the IEEE\/CVF International Conference on Computer Vision. 2021: 13401-13412.","DOI":"10.1109\/ICCV48922.2021.01315"},{"key":"pone.0277463.ref035","unstructured":"Lin J, Yang A, Zhang Y, et al. Interbert: Vision-and-language interaction for multi-modal pretraining. arXiv preprint arXiv:2003.13198, 2020."},{"key":"pone.0277463.ref036","doi-asserted-by":"crossref","unstructured":"Murahari V, Batra D, Parikh D, et al. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. European Conference on Computer Vision. Springer, Cham, 2020: 336-352.","DOI":"10.1007\/978-3-030-58523-5_20"},{"key":"pone.0277463.ref037","article-title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","volume":"32","author":"J Lu","year":"2019","journal-title":"Advances in neural information processing systems"},{"key":"pone.0277463.ref038","doi-asserted-by":"crossref","unstructured":"Zhan X, Wu Y, Dong X, et al. Product1m: Towards weakly supervised instance-level product retrieval via cross-modal pretraining. Proceedings of the IEEE\/CVF International Conference on Computer Vision. 2021: 11782-11791.","DOI":"10.1109\/ICCV48922.2021.01157"},{"key":"pone.0277463.ref039","doi-asserted-by":"crossref","unstructured":"Tsai Y H H, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences. Proceedings of the conference. Association for Computational Linguistics. Meeting. NIH Public Access, 2019, 2019: 6558.","DOI":"10.18653\/v1\/P19-1656"},{"key":"pone.0277463.ref040","unstructured":"Wang G, Tan L, Shang Z, et al. Multimodal Dual Emotion with Fusion of Visual Sentiment for Rumor Detection. arXiv preprint arXiv:2204.11515, 2022."},{"key":"pone.0277463.ref041","doi-asserted-by":"crossref","unstructured":"Azri A, Favre C, Harbi N, et al. Calling to CNN-LSTM for Rumor Detection: A Deep Multi-channel Model for Message Veracity Classification in Microblogs. Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2021: 497-513..","DOI":"10.1007\/978-3-030-86517-7_31"},{"key":"pone.0277463.ref042","doi-asserted-by":"crossref","unstructured":"Kirchknopf A, Slijepcevic D, Zeppelzauer M. Multimodal detection of information disorder from social media. arXiv preprint arXiv:2105.15165, 2021..","DOI":"10.1109\/CBMI50038.2021.9461898"},{"key":"pone.0277463.ref043","doi-asserted-by":"crossref","unstructured":"Wang Y, Ma F, Jin Z, et al. Eann: Event adversarial neural networks for multi-modal fake news detection. Proceedings of the 24th acm sigkdd international conference on knowledge discovery data mining. 2018: 849-857.","DOI":"10.1145\/3219819.3219903"},{"key":"pone.0277463.ref044","doi-asserted-by":"crossref","unstructured":"Khattar D, Goud J S, Gupta M, et al. Mvae: Multimodal variational autoencoder for fake news detection. The world wide web conference. 2019: 2915-2921.","DOI":"10.1145\/3308558.3313552"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0277463","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T14:32:11Z","timestamp":1669300331000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0277463"}},"subtitle":[],"editor":[{"given":"T. Ganesh","family":"Kumar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,11,23]]},"references-count":44,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,11,23]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0277463","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,23]]}}}