{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T15:17:41Z","timestamp":1774451861458,"version":"3.50.1"},"reference-count":30,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T00:00:00Z","timestamp":1708646400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Key Scientific and Technological Project of Henan Province of China","award":["No.232102211013"],"award-info":[{"award-number":["No.232102211013"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Language bias stands as a noteworthy concern in visual question answering (VQA), wherein models tend to rely on spurious correlations between questions and answers for prediction. This prevents the models from effectively generalizing, leading to a decrease in performance. In order to address this bias, we propose a novel modality fusion collaborative de-biasing algorithm (CoD). In our approach, bias is considered as the model\u2019s neglect of information from a particular modality during prediction. We employ a collaborative training approach to facilitate mutual modeling between different modalities, achieving efficient feature fusion and enabling the model to fully leverage multimodal knowledge for prediction. Our experiments on various datasets, including VQA-CP v2, VQA v2, and VQA-VS, using different validation strategies, demonstrate the effectiveness of our approach. Notably, employing a basic baseline model resulted in an accuracy of 60.14% on VQA-CP v2.<\/jats:p>","DOI":"10.3390\/jimaging10030056","type":"journal-article","created":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T10:47:30Z","timestamp":1708685250000},"page":"56","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering"],"prefix":"10.3390","volume":"10","author":[{"given":"Qiwen","family":"Lu","sequence":"first","affiliation":[{"name":"School of Computer and Information Engineering, Henan University, Kaifeng 475001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shengbo","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer and Information Engineering, Henan University, Kaifeng 475001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoke","family":"Zhu","sequence":"additional","affiliation":[{"name":"School of Computer and Information Engineering, Henan University, Kaifeng 475001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,2,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"6807","DOI":"10.1109\/TPAMI.2021.3139957","article-title":"Depth and video segmentation based visual attention for embodied question answering","volume":"45","author":"Luo","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"102750","DOI":"10.1016\/j.ipm.2021.102750","article-title":"Machine learning-based human-robot interaction in its","volume":"59","author":"Wang","year":"2022","journal-title":"Inf. Process. Manag."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Han, X., Wang, S., Su, C., Huang, Q., and Tian, Q. (2021, January 11\u201317). Greedy gradient ensemble for robust visual question answering. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00161"},{"key":"ref_4","first-page":"3784","article-title":"Debiased visual question answering from feature and sample perspectives","volume":"34","author":"Wen","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"103296","DOI":"10.1016\/j.ipm.2023.103296","article-title":"Be flexible! learn to debias by sampling and prompting for robust visual question answering","volume":"60","author":"Liu","year":"2023","journal-title":"Inf. Process. Manag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"85980","DOI":"10.1109\/ACCESS.2023.3304415","article-title":"Overcoming language priors via shuffling language bias for robust visual question answering","volume":"11","author":"Zhao","year":"2023","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Chen, L., Yan, X., Xiao, J., Zhang, H., Pu, S., and Zhuang, Y. (2020, January 13\u201319). Counterfactual samples synthesizing for robust visual question answering. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01081"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Kolling, C., More, M., Gavenski, N., Pooch, E., Parraga, O., and Barros, R. (2022, January 3\u20138). Efficient counterfactual debiasing for visual question answering. Proceedings of the 2022 IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.","DOI":"10.1109\/WACV51458.2022.00263"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhu, X., Mao, Z., Liu, C., Zhang, P., Wang, B., and Zhang, Y. (2020, January 7\u201315). Overcoming language priors with self-supervised learning for visual question answering. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, Yokohama, Japan.","DOI":"10.24963\/ijcai.2020\/151"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shrestha, R., Kafle, K., and Kanan, C. (2020, January 5\u201310). A negative case analysis of visual grounding methods for vqa. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.727"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Baltrusaitis","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Agrawal, A., Batra, D., and Parikh, D. (2016, January 1\u20135). Analyzing the behavior of visual question answering models. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1203"},{"key":"ref_13","first-page":"841","article-title":"Rubi: Reducing unimodal biases for visual question answering","volume":"32","author":"Dancette","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Clark, C., Yatskar, M., and Zettlemoyer, L. (2019, January 3\u20137). Don\u2019t take the easy way out: Ensemble based methods for avoiding known dataset biases. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1418"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X., and Wen, J. (2021, January 20\u201325). Counterfactual vqa: A cause-effect look at language bias. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01251"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"9789","DOI":"10.1109\/TPAMI.2023.3240337","article-title":"General greedy de-bias learning","volume":"45","author":"Han","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Guo, Y., Cheng, Z., Nie, L., Liu, Y., Wang, Y., and Kankanhalli, M. (2019, January 21\u201325). Quantifying and alleviating the language prior problem in visual question answering. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France.","DOI":"10.1145\/3331184.3331186"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yuan, D., Wang, L., Wu, Q., Meng, F., Ngan, K.N., and Xu, L. (2022). Language bias-driven self-knowledge distillation with generalization uncertainty for reducing language bias in visual question answering. Appl. Sci., 12.","DOI":"10.3390\/app12157588"},{"key":"ref_19","first-page":"1541","article-title":"Overcoming language priors in visual question answering with adversarial regularization","volume":"31","author":"Ramakrishnan","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_20","unstructured":"Oord, A., Li, Y., and Vinyals, O. (2019). Representation learning with contrastive predictive coding. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Si, Q., Lin, Z., Zheng, M.Y., Fu, P., and Wang, W. (2021, January 1\u20136). Check it again: Progressive visual question answering via visual entailment. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual Event.","DOI":"10.18653\/v1\/2021.acl-long.317"},{"key":"ref_22","unstructured":"Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020, January 13\u201318). A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, PMLR, Virtual Event."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., and Parikh, D. (2019, January 27\u201328). Taking a hint: Leveraging explanations to make vision and language models more grounded. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00268"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Liang, Z., Hu, H., and Zhu, J. (2021, January 11\u201315). Lpf: A language-prior feedback objective function for de-biased visual question answering. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Montreal, QC, Canada.","DOI":"10.1145\/3404835.3462981"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/35.41402","article-title":"Integration of acoustic and visual speech signals using neural networks","volume":"27","author":"Yuhas","year":"1989","journal-title":"IEEE Commun. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Arora, R., and Livescu, K. (2013, January 26\u201331). Multi-view cca-based acoustic features for phonetic recognition across speakers and domains. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6639047"},{"key":"ref_27","first-page":"2121","article-title":"Devise: A deep visual-semantic embedding model","volume":"26","author":"Frome","year":"2013","journal-title":"Neural Inf. Process. Syst. Inf. Process. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bazi, Y., Rahhal, M.M.A., Bashmal, L., and Zuair, M. (2023). Vision\u2013Language model for visual question answering in medical imagery. Bioengineering, 10.","DOI":"10.3390\/bioengineering10030380"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhu, H., Togo, R., Ogawa, T., and Haseyama, M. (2023). Multimodal natural language explanation generation for visual question answering based on multiple reference data. Electronics, 12.","DOI":"10.3390\/electronics12102183"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Jiang, L., and Meng, Z. (2023). Knowledge-based visual question answering using multi-modal semantic graph. Electronics, 12.","DOI":"10.3390\/electronics12061390"}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/3\/56\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:04:01Z","timestamp":1760105041000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/3\/56"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,23]]},"references-count":30,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["jimaging10030056"],"URL":"https:\/\/doi.org\/10.3390\/jimaging10030056","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,23]]}}}