{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T05:01:11Z","timestamp":1781586071222,"version":"3.54.5"},"reference-count":196,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2025,3,5]],"date-time":"2025-03-05T00:00:00Z","timestamp":1741132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:p>Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.<\/jats:p>","DOI":"10.1145\/3711680","type":"journal-article","created":{"date-parts":[[2025,1,31]],"date-time":"2025-01-31T11:07:27Z","timestamp":1738321647000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-5764-1398","authenticated-orcid":false,"given":"Jiayi","family":"Kuang","sequence":"first","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3220-904X","authenticated-orcid":false,"given":"Ying","family":"Shen","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0594-4900","authenticated-orcid":false,"given":"Jingyou","family":"Xie","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9714-0434","authenticated-orcid":false,"given":"Haohao","family":"Luo","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9669-1966","authenticated-orcid":false,"given":"Zhe","family":"Xu","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7174-2638","authenticated-orcid":false,"given":"Ronghao","family":"Li","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7571-6722","authenticated-orcid":false,"given":"Yinghui","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8589-9313","authenticated-orcid":false,"given":"Xianfeng","family":"Cheng","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6919-7831","authenticated-orcid":false,"given":"Xika","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Worcester Polytechnic Institute, Worcester, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7550-1737","authenticated-orcid":false,"given":"Yu","family":"Han","sequence":"additional","affiliation":[{"name":"Sun Yat-Sen University, Shenzhen China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,3,5]]},"reference":[{"issue":"6","key":"e_1_3_2_2_2","article-title":"VQA-Med: Overview of the medical visual question answering task at ImageCLEF 2019.","volume":"2","author":"Abacha Asma Ben","year":"2019","unstructured":"Asma Ben Abacha, Sadid A. Hasan, Vivek V. Datla, Joey Liu, Dina Demner-Fushman, and Henning M\u00fcller. 2019. VQA-Med: Overview of the medical visual question answering task at ImageCLEF 2019. CLEF (Working Notes) 2, 6 (2019), 1\u201311.","journal-title":"CLEF (Working Notes)"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00522"},{"key":"e_1_3_2_4_2","first-page":"23716","article-title":"Flamingo: A visual language model for few-shot learning","volume":"35","author":"Alayrac Jean-Baptiste","year":"2022","unstructured":"Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et\u00a0al. 2022. Flamingo: A visual language model for few-shot learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems 35 (2022), 23716\u201323736.","journal-title":"Proceedings of the 36th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00636"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.279"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-76298-0_52"},{"key":"e_1_3_2_8_2","unstructured":"Jinze Bai Shuai Bai Shusheng Yang Shijie Wang Sinan Tan Peng Wang Junyang Lin Chang Zhou and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding localization text reading and beyond. arXiv preprint arXiv:2308.12966 2023."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.5555\/3491440.3491962"},{"key":"e_1_3_2_10_2","first-page":"65","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. 65\u201372."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1016\/j.patrec.2021.09.008","article-title":"Visual question answering: Which investigated applications?","volume":"151","author":"Barra Silvio","year":"2021","unstructured":"Silvio Barra, Carmen Bisogni, Maria De Marsico, and Stefano Ricciardi. 2021. Visual question answering: Which investigated applications? Pattern Recogn. Lett. 151, C (Nov2021), 325\u2013331.","journal-title":"Pattern Recogn. Lett."},{"key":"e_1_3_2_12_2","article-title":"Translating embeddings for modeling multi-relational data","volume":"26","author":"Bordes Antoine","year":"2013","unstructured":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems 26 (2013).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_13_2","volume-title":"Computer Vision - ECCV 2024-18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part L, volume 15108 of Lecture Notes in Computer Science.","author":"Cai Rizhao","year":"2024","unstructured":"Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Yaohang Li, Xing Luo, Chenyu Yi, and Alex C. Kot. 2024. Benchlmm: Benchmarking cross-style visual capability of large multimodal models. In Computer Vision - ECCV 2024-18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part L, volume 15108 of Lecture Notes in Computer Science. 340\u2013358."},{"key":"e_1_3_2_14_2","article-title":"Knowledge-routed visual question reasoning: Challenges for deep representation embedding","author":"Cao Qingxing","year":"2021","unstructured":"Qingxing Cao, Bailin Li, Xiaodan Liang, Keze Wang, and Liang Lin. 2021. Knowledge-routed visual question reasoning: Challenges for deep representation embedding. IEEE Transactions on Neural Networks and Learning Systems (2021).","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"e_1_3_2_15_2","unstructured":"Feilong Chen Minglun Han Haozhi Zhao Qingyang Zhang Jing Shi Shuang Xu and Bo Xu. 2023. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160 2023."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Keqin Chen Zhao Zhang Weili Zeng Richong Zhang Feng Zhu and Rui Zhao. 2023. Shikra: Unleashing multimodal LLM\u2019s referential dialogue magic. 10.48550\/arXiv.2306.15195","DOI":"10.48550\/arXiv.2306.15195"},{"key":"e_1_3_2_17_2","unstructured":"Xi Chen Xiao Wang Lucas Beyer Alexander Kolesnikov Jialin Wu Paul Voigtlaender Basil Mustafa Sebastian Goodman Ibrahim Alabdulmohsin Piotr Padlewski et al.. 2023. Pali-3 vision language models: Smaller faster stronger."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58577-8_7"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","unstructured":"Zhe Chen Weiyun Wang Hao Tian Shenglong Ye Zhangwei Gao Erfei Cui Wenwen Tong Kongzhi Hu Jiapeng Luo Zheng Ma et\u00a0al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.CoRR abs\/2404.16821. 10.48550\/arXiv.2404.16821","DOI":"10.48550\/arXiv.2404.16821"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.02283"},{"key":"e_1_3_2_21_2","unstructured":"Chenhang Cui Yiyang Zhou Xinyu Yang Shirley Wu Linjun Zhang James Zou and Huaxiu Yao. 2023. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287 2023."},{"key":"e_1_3_2_22_2","volume-title":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10\u201316, 2023","author":"Dai Wenliang","year":"2023","unstructured":"Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10\u201316, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.)."},{"key":"e_1_3_2_23_2","unstructured":"Ana Claudia Akemi Matsuki de Faria Felype de Castro Bastos Jose Victor Nogueira Alves da Silva Vitor Lopes Fabris Valeska de Sousa Uchoa D\u00e9cio Gon\u00e7alves de Aguiar Neto and Claudio Filipi Goncalves dos Santos. 2023. Visual question answering: A survey on techniques and common trends in recent literature. CoRR abs\/2305.11033 2023."},{"key":"e_1_3_2_24_2","first-page":"10088","volume-title":"Advances in Neural Information Processing Systems","author":"Dettmers Tim","year":"2023","unstructured":"Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems, A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 10088\u201310115."},{"key":"e_1_3_2_25_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 1 (Long and Short Papers). 4171\u20134186."},{"key":"e_1_3_2_26_2","first-page":"585","volume-title":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","author":"Ding Yihao","year":"2023","unstructured":"Yihao Ding, Siwen Luo, Hyunsuk Chung, and Soyeon Caren Han. 2023. VQA: A new dataset for real-world VQA on PDF documents. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 585\u2013601."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00503"},{"key":"e_1_3_2_28_2","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024. Association for Computational Linguistics","author":"Dong Qingxiu","year":"2023","unstructured":"Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey for in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024. Association for Computational Linguistics, 1107\u20131128."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623623"},{"key":"e_1_3_2_30_2","volume-title":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net","author":"Dosovitskiy Alexey","year":"2021","unstructured":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et\u00a0al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. https:\/\/openreview.net\/forum?id=YicbFdNTTy"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01763"},{"key":"e_1_3_2_32_2","series-title":"Proceedings of Machine Learning Research","first-page":"8469","volume-title":"International Conference on Machine Learning, ICML 2023, 23\u201329 July 2023, Honolulu, Hawaii, USA","volume":"202","author":"Driess Danny","year":"2023","unstructured":"Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. PaLM-E: An embodied multimodal language model. In International Conference on Machine Learning, ICML 2023, 23\u201329 July 2023, Honolulu, Hawaii, USA(Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 8469\u20138488."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.5555\/1763974.1764031"},{"key":"e_1_3_2_34_2","unstructured":"Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang Xu Lin Jinrui Yang Xiawu Zheng Ke Li Xing Sun Yunsheng Wu and Rongrong Ji. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv:2306.13394. Retrieved from https:\/\/arxiv.org\/abs\/2306.13394"},{"key":"e_1_3_2_35_2","article-title":"A challenger to gpt-4v? Early explorations of gemini in visual expertise","author":"Fu Chaoyou","year":"2023","unstructured":"Chaoyou Fu, Renrui Zhang, Haojia Lin, Zihan Wang, Timin Gao, Yongdong Luo, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, et\u00a0al. 2023. A challenger to gpt-4v? Early explorations of gemini in visual expertise. arXiv preprint arXiv:2312.12436 (2023).","journal-title":"arXiv preprint arXiv:2312.12436"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1044"},{"key":"e_1_3_2_37_2","first-page":"6616","article-title":"Large-scale adversarial training for vision-and-language representation learning","volume":"33","author":"Gan Zhe","year":"2020","unstructured":"Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems 33 (2020), 6616\u20136628.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_38_2","first-page":"5067","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Gao Feng","year":"2022","unstructured":"Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. 2022. Transform-retrieve-generate: Natural language-centric outside-knowledge visual question answering. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 5067\u20135077."},{"key":"e_1_3_2_39_2","article-title":"Are you talking to a machine? Dataset and methods for multilingual image question","volume":"28","author":"Gao Haoyuan","year":"2015","unstructured":"Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. 2015. Are you talking to a machine? Dataset and methods for multilingual image question. Advances in Neural Information Processing Systems 28 (2015).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_40_2","article-title":"Lora: A logical reasoning augmented dataset for visual question answering","volume":"36","author":"Gao Jingying","year":"2024","unstructured":"Jingying Gao, Qi Wu, Alan Blair, and Maurice Pagnucco. 2024. Lora: A logical reasoning augmented dataset for visual question answering. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_41_2","first-page":"469","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201918)","author":"Gao Peng","year":"2018","unstructured":"Peng Gao, Hongsheng Li, Shuang Li, Pan Lu, Yikang Li, Steven C. H. Hoi, and Xiaogang Wang. 2018. Question-guided hybrid convolution for visual question answering. In Proceedings of the European Conference on Computer Vision (ECCV\u201918). 469\u2013485."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.44"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.670"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24797-2_4"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01046"},{"key":"e_1_3_2_46_2","first-page":"14953","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201923)","author":"Gupta Tanmay","year":"2023","unstructured":"Tanmay Gupta and Aniruddha Kembhavi. 2023. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201923). 14953\u201314962."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00380"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_49_2","article-title":"Hypergraph transformer: Weakly-supervised multi-hop reasoning for knowledge-based visual question answering","author":"Heo Yu-Jung","year":"2022","unstructured":"Yu-Jung Heo, Eun-Sol Kim, Woo Suk Choi, and Byoung-Tak Zhang. 2022. Hypergraph transformer: Weakly-supervised multi-hop reasoning for knowledge-based visual question answering. arXiv preprint arXiv:2204.10448 (2022).","journal-title":"arXiv preprint arXiv:2204.10448"},{"key":"e_1_3_2_50_2","first-page":"204","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\u201310, 2023","author":"Himakunthala Vaishnavi","year":"2023","unstructured":"Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, and William Yang Wang. 2023. Let\u2019s think frame by frame with VIP: A video infilling and prediction dataset for evaluating video chain-of-thought. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\u201310, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 204\u2013219."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.01039"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.642"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01278"},{"key":"e_1_3_2_54_2","article-title":"Pixel-bert: Aligning image pixels with text by deep multi-modal transformers","author":"Huang Zhicheng","year":"2020","unstructured":"Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849 (2020).","journal-title":"arXiv preprint arXiv:2004.00849"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00686"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2024.102270"},{"key":"e_1_3_2_57_2","article-title":"GPT-4o: The cutting-edge advancement in multimodal LLM","author":"Islam Raisa","year":"2024","unstructured":"Raisa Islam and Owana Marzia Moushi. 2024. GPT-4o: The cutting-edge advancement in multimodal LLM. Authorea Preprints (2024).","journal-title":"Authorea Preprints"},{"key":"e_1_3_2_58_2","article-title":"Opt-iml: Scaling language model instruction meta learning through the lens of generalization","author":"Iyer Srinivasan","year":"2022","unstructured":"Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et\u00a0al. 2022. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017 (2022).","journal-title":"arXiv preprint arXiv:2212.12017"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01028"},{"key":"e_1_3_2_60_2","article-title":"LLMs can find mathematical reasoning mistakes by pedagogical chain-of-thought","volume":"2405","author":"Jiang Zhuoxuan","year":"2024","unstructured":"Zhuoxuan Jiang, Haoyuan Peng, Shanshan Feng, Fan Li, and Dongsheng Li. 2024. LLMs can find mathematical reasoning mistakes by pedagogical chain-of-thought. CoRR abs\/2405.06705 (2024).","journal-title":"CoRR"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.215"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.494"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.06.005"},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP\u201920)","author":"Karpukhin Vladimir","year":"2020","unstructured":"Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP\u201920). 6769\u20136781."},{"key":"e_1_3_2_65_2","first-page":"5583","volume-title":"International Conference on Machine Learning","author":"Kim Wonjae","year":"2021","unstructured":"Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning. PMLR, 5583\u20135594."},{"key":"e_1_3_2_66_2","article-title":"Generating images with multimodal language models","volume":"36","author":"Koh Jing Yu","year":"2024","unstructured":"Jing Yu Koh, Daniel Fried, and Russ R. Salakhutdinov. 2024. Generating images with multimodal language models. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0981-7"},{"key":"e_1_3_2_68_2","unstructured":"John Lafferty Andrew McCallum and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. (2001)."},{"key":"e_1_3_2_69_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1007\/978-981-97-3076-6_18","volume-title":"New Frontiers in Artificial Intelligence - JSAI International Symposium on Artificial Intelligence, JSAI-isAI 2024, Hamamatsu, Japan, May 28\u201329, 2024, Proceedings","volume":"14741","author":"Le Nguyen-Khang","year":"2024","unstructured":"Nguyen-Khang Le, Dieu-Hien Nguyen, Dinh-Truong Do, Chau Nguyen, and Le Minh Nguyen. 2024. Vietnamese elementary math reasoning using large language model with refined translation and dense-retrieved chain-of-thought. In New Frontiers in Artificial Intelligence - JSAI International Symposium on Artificial Intelligence, JSAI-isAI 2024, Hamamatsu, Japan, May 28\u201329, 2024, Proceedings(Lecture Notes in Computer Science, Vol. 14741), Toyotaro Suzumura and Mayumi Bono (Eds.). Springer, 260\u2013268."},{"key":"e_1_3_2_70_2","first-page":"1188","volume-title":"International Conference on Machine Learning","author":"Le Quoc","year":"2014","unstructured":"Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning. PMLR, 1188\u20131196."},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531753"},{"key":"e_1_3_2_72_2","article-title":"SEED-Bench-2-Plus: Benchmarking multimodal large language models with text-rich visual comprehension","author":"Li Bohao","year":"2024","unstructured":"Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. 2024. SEED-Bench-2-Plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790 (2024).","journal-title":"arXiv preprint arXiv:2404.16790"},{"key":"e_1_3_2_73_2","volume-title":"CVPR","author":"Li Bohao","year":"2024","unstructured":"Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal llms with generative comprehension. In CVPR."},{"key":"e_1_3_2_74_2","article-title":"Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing","volume":"36","author":"Li Dongxu","year":"2024","unstructured":"Dongxu Li, Junnan Li, and Steven Hoi. 2024. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6795"},{"key":"e_1_3_2_76_2","article-title":"Incorporating external knowledge to answer open-domain visual questions with dynamic memory networks","author":"Li Guohao","year":"2017","unstructured":"Guohao Li, Hang Su, and Wenwu Zhu. 2017. Incorporating external knowledge to answer open-domain visual questions with dynamic memory networks. arXiv preprint arXiv:1712.00733 (2017).","journal-title":"arXiv preprint arXiv:1712.00733"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394171.3413943"},{"key":"e_1_3_2_78_2","first-page":"19730","volume-title":"International Conference on Machine Learning","author":"Li Junnan","year":"2023","unstructured":"Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning. PMLR, 19730\u201319742."},{"key":"e_1_3_2_79_2","article-title":"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation","author":"Li Junnan","year":"2022","unstructured":"Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086 (2022).","journal-title":"arXiv preprint arXiv:2201.12086"},{"key":"e_1_3_2_80_2","first-page":"9694","article-title":"Align before fuse: Vision and language representation learning with momentum distillation","volume":"34","author":"Li Junnan","year":"2021","unstructured":"Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems 34 (2021), 9694\u20139705.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.01041"},{"key":"e_1_3_2_82_2","article-title":"Dynamic key-value memory enhanced multi-step graph reasoning for knowledge-based visual question answering","author":"Li Mingxiao","year":"2022","unstructured":"Mingxiao Li and Marie-Francine Moens. 2022. Dynamic key-value memory enhanced multi-step graph reasoning for knowledge-based visual question answering. arXiv preprint arXiv:2203.02985 (2022).","journal-title":"arXiv preprint arXiv:2203.02985"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3489142"},{"key":"e_1_3_2_84_2","article-title":"Visual question answering with question representation update (qru)","volume":"29","author":"Li Ruiyu","year":"2016","unstructured":"Ruiyu Li and Jiaya Jia. 2016. Visual question answering with question representation update (qru). Advances in Neural Information Processing Systems 29 (2016).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_85_2","article-title":"Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning","author":"Li Wei","year":"2020","unstructured":"Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2020. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. arXiv preprint arXiv:2012.15409 (2020).","journal-title":"arXiv preprint arXiv:2012.15409"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58577-8_8"},{"key":"e_1_3_2_87_2","article-title":"MLLM-SR: Conversational symbolic regression base multi-modal large language models","author":"Li Yanjie","year":"2024","unstructured":"Yanjie Li, Weijun Li, Lina Yu, Min Wu, Jingyi Liu, Wenqiang Li, Shu Wei, and Yusong Deng. 2024. MLLM-SR: Conversational symbolic regression base multi-modal large language models. arXiv preprint arXiv:2406.05410 (2024).","journal-title":"arXiv preprint arXiv:2406.05410"},{"key":"e_1_3_2_88_2","article-title":"Towards real-world writing assistance: A chinese character checking benchmark with faked and misspelled characters","author":"Li Yinghui","year":"2023","unstructured":"Yinghui Li, Zishan Xu, Shaoshen Chen, Haojing Huang, Yangning Li, Yong Jiang, Zhongli Li, Qingyu Zhou, Hai-Tao Zheng, and Ying Shen. 2023. Towards real-world writing assistance: A chinese character checking benchmark with faked and misspelled characters. arXiv preprint arXiv:2311.11268 (2023).","journal-title":"arXiv preprint arXiv:2311.11268"},{"key":"e_1_3_2_89_2","volume-title":"The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track","author":"Li Yinghui","year":"2024","unstructured":"Yinghui Li, Qingyu Zhou, Yuanzhen Luo, Shirong Ma, Yangning Li, Hai-Tao Zheng, Xuming Hu, and S. Yu Philip. 2024. When LLMs meet cunning texts: A fallacy understanding benchmark for large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_90_2","article-title":"EAGLE: Elevating geometric reasoning through LLM-empowered visual instruction tuning","author":"Li Zhihao","year":"2024","unstructured":"Zhihao Li, Yao Du, Yang Liu, Yan Zhang, Yufang Liu, Mengdi Zhang, and Xunliang Cai. 2024. EAGLE: Elevating geometric reasoning through LLM-empowered visual instruction tuning. arXiv preprint arXiv:2408.11397 (2024).","journal-title":"arXiv preprint arXiv:2408.11397"},{"key":"e_1_3_2_91_2","article-title":"LRTA: A transparent neural-symbolic reasoning framework with modular supervision for visual question answering","author":"Liang Weixin","year":"2020","unstructured":"Weixin Liang, Feiyang Niu, Aishwarya Reganti, Govind Thattai, and Gokhan Tur. 2020. LRTA: A transparent neural-symbolic reasoning framework with modular supervision for visual question answering. arXiv preprint arXiv:2011.10731 (2020).","journal-title":"arXiv preprint arXiv:2011.10731"},{"key":"e_1_3_2_92_2","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. 74\u201381."},{"key":"e_1_3_2_93_2","article-title":"Hallusionbench: You see what you think? or you think what you see? An image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models","author":"Liu Fuxiao","year":"2023","unstructured":"Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023. Hallusionbench: You see what you think? or you think what you see? An image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint arXiv:2310.14566 (2023).","journal-title":"arXiv preprint arXiv:2310.14566"},{"key":"e_1_3_2_94_2","article-title":"Visual instruction tuning","volume":"36","author":"Liu Haotian","year":"2024","unstructured":"Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:BTTJ.0000047600.45421.6d"},{"issue":"7","key":"e_1_3_2_96_2","article-title":"Are we ready for a new paradigm shift? A survey on visual deep mlp","volume":"3","author":"Liu Ruiyang","year":"2022","unstructured":"Ruiyang Liu, Yinghui Li, Linmi Tao, Dun Liang, and Hai-Tao Zheng. 2022. Are we ready for a new paradigm shift? A survey on visual deep mlp. Patterns 3, 7 (2022), 1\u201325.","journal-title":"Patterns"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"e_1_3_2_98_2","article-title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","volume":"32","author":"Lu Jiasen","year":"2019","unstructured":"Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01045"},{"key":"e_1_3_2_100_2","article-title":"Hierarchical question-image co-attention for visual question answering","volume":"29","author":"Lu Jiasen","year":"2016","unstructured":"Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems 29 (2016).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_101_2","volume-title":"ICLR","author":"Lu Pan","year":"2024","unstructured":"Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. In ICLR."},{"key":"e_1_3_2_102_2","first-page":"1880","volume-title":"Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","author":"Lu Pan","year":"2018","unstructured":"Pan Lu, Lei Ji, Wei Zhang, Nan Duan, Ming Zhou, and Jianyong Wang. 2018. R-VQA: Learning visual relation facts with semantic attention for visual question answering. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1880\u20131889."},{"key":"e_1_3_2_103_2","volume-title":"Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022","author":"Lu Pan","year":"2022","unstructured":"Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.)."},{"key":"e_1_3_2_104_2","article-title":"Chameleon: Plug-and-play compositional reasoning with large language models","volume":"36","author":"Lu Pan","year":"2024","unstructured":"Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2024. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_105_2","volume-title":"NeurIPS Datasets and Benchmarks","author":"Lu Pan","year":"2021","unstructured":"Pan Lu, Liang Qiu, Jiaqi Chen, Tanglin Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurIPS Datasets and Benchmarks."},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.518"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.556"},{"key":"e_1_3_2_108_2","first-page":"6975","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Ma Chao","year":"2018","unstructured":"Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. 2018. Visual question answering with memory-augmented networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6975\u20136984."},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2024.3366154"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.5555\/3016387.3016405"},{"key":"e_1_3_2_111_2","article-title":"A multi-world approach to question answering about real-world scenes based on uncertain input","volume":"27","author":"Malinowski Mateusz","year":"2014","unstructured":"Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. Advances in Neural Information Processing Systems 27 (2014).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00331"},{"key":"e_1_3_2_113_2","first-page":"1697","volume-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision","author":"Mathew Minesh","year":"2022","unstructured":"Minesh Mathew, Viraj Bagal, Rub\u00e8n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. InfographicVQA. In Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision. 1697\u20131706."},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00225"},{"key":"e_1_3_2_115_2","article-title":"Distributed representations of words and phrases and their compositionality","volume":"26","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26 (2013).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.232"},{"key":"e_1_3_2_117_2","article-title":"Out of the box: Reasoning with graph convolution nets for factual visual question answering","volume":"31","author":"Narasimhan Medhini","year":"2018","unstructured":"Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. 2018. Out of the box: Reasoning with graph convolution nets for factual visual question answering. Advances in Neural Information Processing Systems 31 (2018).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_118_2","first-page":"451","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201918)","author":"Narasimhan Medhini","year":"2018","unstructured":"Medhini Narasimhan and Alexander G. Schwing. 2018. Straight to the facts: Learning knowledge base retrieval for factual visual question answering. In Proceedings of the European Conference on Computer Vision (ECCV\u201918). 451\u2013468."},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00637"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.11"},{"key":"e_1_3_2_121_2","article-title":"Learning conditioned graph structures for interpretable visual question answering","volume":"31","author":"Norcliffe-Brown Will","year":"2018","unstructured":"Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. 2018. Learning conditioned graph structures for interpretable visual question answering. Advances in Neural Information Processing Systems 31 (2018).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_122_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_3_2_123_2","article-title":"TALM: Tool augmented language models","volume":"2205","author":"Parisi Aaron","year":"2022","unstructured":"Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. TALM: Tool augmented language models. CoRR abs\/2205.12255 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_124_2","first-page":"339","volume-title":"International Conference on Pattern Recognition","author":"Patel Devshree","year":"2021","unstructured":"Devshree Patel, Ratnam Parikh, and Yesha Shastri. 2021. Recent advances in video question answering: A review of datasets and methods. In International Conference on Pattern Recognition. Springer, 339\u2013356."},{"key":"e_1_3_2_125_2","first-page":"321","volume-title":"International Conference on Artificial Intelligence in Education","author":"J\u00fanior Cleon Pereira","year":"2024","unstructured":"Cleon Pereira J\u00fanior, Luiz Rodrigues, Newarney Costa, Valmir Macario Filho, and Rafael Mello. 2024. Can VLM understand children\u2019s handwriting? An analysis on handwritten mathematical equation recognition. In International Conference on Artificial Intelligence in Education. Springer, 321\u2013328."},{"key":"e_1_3_2_126_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462987"},{"key":"e_1_3_2_127_2","first-page":"8748","volume-title":"International Conference on Machine Learning","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et\u00a0al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748\u20138763."},{"key":"e_1_3_2_128_2","article-title":"Seeing is knowing! fact-based visual question answering using knowledge graph embeddings","author":"Ramnath Kiran","year":"2020","unstructured":"Kiran Ramnath and Mark Hasegawa-Johnson. 2020. Seeing is knowing! fact-based visual question answering using knowledge graph embeddings. arXiv preprint arXiv:2012.15484 (2020).","journal-title":"arXiv preprint arXiv:2012.15484"},{"key":"e_1_3_2_129_2","article-title":"Exploring models and data for image question answering","volume":"28","author":"Ren Mengye","year":"2015","unstructured":"Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Exploring models and data for image question answering. Advances in Neural Information Processing Systems 28 (2015).","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"e_1_3_2_130_2","first-page":"5","article-title":"Image question answering: A visual semantic embedding model and a new dataset","volume":"1","author":"Ren Mengye","year":"2015","unstructured":"Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Image question answering: A visual semantic embedding model and a new dataset. Proc. Advances in Neural Inf. Process. Syst 1, 2 (2015), 5.","journal-title":"Proc. Advances in Neural Inf. Process. Syst"},{"key":"e_1_3_2_131_2","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume":"28","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 28 (2015).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_132_2","article-title":"Charting new territories: Exploring the geographic and geospatial capabilities of multimodal llms","author":"Roberts Jonathan","year":"2023","unstructured":"Jonathan Roberts, Timo L\u00fcddecke, Rehan Sheikh, Kai Han, and Samuel Albanie. 2023. Charting new territories: Exploring the geographic and geospatial capabilities of multimodal llms. arXiv preprint arXiv:2311.14656 (2023).","journal-title":"arXiv preprint arXiv:2311.14656"},{"key":"e_1_3_2_133_2","article-title":"Visual chain of thought: Bridging logical gaps with multimodal infillings","volume":"2305","author":"Rose Daniel","year":"2023","unstructured":"Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. 2023. Visual chain of thought: Bridging logical gaps with multimodal infillings. CoRR abs\/2305.02317 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_134_2","volume-title":"ICLR 2022-Tenth International Conference on Learning Representations","author":"Sanh Victor","year":"2022","unstructured":"Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et\u00a0al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representations."},{"key":"e_1_3_2_135_2","article-title":"High-order attention models for visual question answering","volume":"30","author":"Schwartz Idan","year":"2017","unstructured":"Idan Schwartz, Alexander Schwing, and Tamir Hazan. 2017. High-order attention models for visual question answering. Advances in Neural Information Processing Systems 30 (2017).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00681"},{"key":"e_1_3_2_137_2","first-page":"8876","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"33","author":"Shah Sanket","year":"2019","unstructured":"Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8876\u20138884."},{"key":"e_1_3_2_138_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01438"},{"key":"e_1_3_2_139_2","article-title":"Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face","volume":"36","author":"Shen Yongliang","year":"2024","unstructured":"Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2024. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_140_2","article-title":"Knowledge-based reasoning network for relation detection","author":"Shen Ying","year":"2023","unstructured":"Ying Shen, Min Yang, Yaliang Li, Dong Wang, Haitao Zheng, and Daoyuan Chen. 2023. Knowledge-based reasoning network for relation detection. IEEE Trans. Neural Networks Learn. Syst., 34, 8 (2023), 5051\u20135063. (2021).","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"e_1_3_2_141_2","first-page":"8376","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Shi Jiaxin","year":"2019","unstructured":"Jiaxin Shi, Hanwang Zhang, and Juanzi Li. 2019. Explainable and explicit visual reasoning over scene graphs. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 8376\u20138384."},{"key":"e_1_3_2_142_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.499"},{"key":"e_1_3_2_143_2","article-title":"Beyond task performance: Evaluating and reducing the flaws of large multimodal models with in-context learning","author":"Shukor Mustafa","year":"2023","unstructured":"Mustafa Shukor, Alexandre Rame, Corentin Dancette, and Matthieu Cord. 2023. Beyond task performance: Evaluating and reducing the flaws of large multimodal models with in-context learning. arXiv preprint arXiv:2310.00647 (2023).","journal-title":"arXiv preprint arXiv:2310.00647"},{"key":"e_1_3_2_144_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33715-4_54"},{"key":"e_1_3_2_145_2","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan Karen","year":"2014","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).","journal-title":"arXiv preprint arXiv:1409.1556"},{"key":"e_1_3_2_146_2","first-page":"778","volume-title":"2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS\u201923)","author":"Singh Harsimran Jit","year":"2023","unstructured":"Harsimran Jit Singh, Gourav Bathla, Munish Mehta, Gunjan Chhabra, and Pardeep Singh. 2023. Visual questions answering developments, applications, datasets and opportunities: A state-of-the-art survey. In 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS\u201923). 778\u2013785."},{"key":"e_1_3_2_147_2","first-page":"75","volume-title":"International Conference on Computer Vision and Image Processing","author":"Srivastava Yash","year":"2021","unstructured":"Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee. 2021. Visual question answering using deep learning: A survey and performance analysis. In International Conference on Computer Vision and Image Processing. Springer, 75\u201386."},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2022.3148210"},{"key":"e_1_3_2_149_2","article-title":"Vl-bert: Pre-training of generic visual-linguistic representations","author":"Su Weijie","year":"2019","unstructured":"Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019).","journal-title":"arXiv preprint arXiv:1908.08530"},{"key":"e_1_3_2_150_2","first-page":"7736","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Su Zhou","year":"2018","unstructured":"Zhou Su, Chen Zhu, Yinpeng Dong, Dongqi Cai, Yurong Chen, and Jianguo Li. 2018. Learning visual knowledge memory networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7736\u20137745."},{"key":"e_1_3_2_151_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_3_2_152_2","article-title":"Lxmert: Learning cross-modality encoder representations from transformers","author":"Tan Hao","year":"2019","unstructured":"Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 (2019).","journal-title":"arXiv preprint arXiv:1908.07490"},{"key":"e_1_3_2_153_2","article-title":"Gemini: A family of highly capable multimodal models","author":"Team Gemini","year":"2023","unstructured":"Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, et\u00a0al. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023).","journal-title":"arXiv preprint arXiv:2312.11805"},{"key":"e_1_3_2_154_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.344"},{"key":"e_1_3_2_155_2","first-page":"951","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\u201311, 2022","author":"Tiong Anthony Meng Huat","year":"2022","unstructured":"Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven C. H. Hoi. 2022. Plug-and-play VQA: Zero-shot VQA by conjoining large pretrained models with zero training. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\u201311, 2022, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 951\u2013967."},{"key":"e_1_3_2_156_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.510"},{"key":"e_1_3_2_157_2","first-page":"200","article-title":"Multimodal few-shot learning with frozen language models","volume":"34","author":"Tsimpoukelli Maria","year":"2021","unstructured":"Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, S. M. Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34 (2021), 200\u2013212.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_158_2","article-title":"Attention is all you need","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. 5998\u20136008.","journal-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA."},{"key":"e_1_3_2_159_2","doi-asserted-by":"publisher","DOI":"10.5555\/3171642.3171825"},{"key":"e_1_3_2_160_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2017.2754246"},{"key":"e_1_3_2_161_2","article-title":"Visionllm: Large language model is also an open-ended decoder for vision-centric tasks","volume":"36","author":"Wang Wenhai","year":"2024","unstructured":"Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et\u00a0al. 2024. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_162_2","volume-title":"International Conference on Learning Representations","author":"Wei Jason","year":"2021","unstructured":"Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. In International Conference on Learning Representations."},{"key":"e_1_3_2_163_2","article-title":"Chain of reasoning for visual question answering","author":"Wu Chenfei","year":"2018","unstructured":"Chenfei Wu, Jinlai Liu, Xiaojie Wang, and Xuan Dong. 2018. Chain of reasoning for visual question answering. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr\u00e9al, Canada. 273\u2013283.","journal-title":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr\u00e9al, Canada."},{"key":"e_1_3_2_164_2","article-title":"Visual ChatGPT: Talking, drawing and editing with visual foundation models","volume":"2303","author":"Wu Chenfei","year":"2023","unstructured":"Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual ChatGPT: Talking, drawing and editing with visual foundation models. CoRR abs\/2303.04671 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_165_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2017.05.001"},{"key":"e_1_3_2_166_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.500"},{"key":"e_1_3_2_167_2","article-title":"Symbol-LLM: Leverage language models for symbolic system in visual human activity reasoning","author":"Wu Xiaoqian","year":"2023","unstructured":"Xiaoqian Wu, Yong-Lu Li, Jianhua Sun, and Cewu Lu. 2023. Symbol-LLM: Leverage language models for symbolic system in visual human activity reasoning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems NeurIPS 2023.http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/5edb57c05c81d04beb716ef1d542fe9e-Abstract-Conference.html","journal-title":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems NeurIPS 2023."},{"key":"e_1_3_2_168_2","article-title":"Logicvista: Multimodal llm logical reasoning benchmark in visual contexts","author":"Xiao Yijia","year":"2024","unstructured":"Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. 2024. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973 (2024).","journal-title":"arXiv preprint arXiv:2407.04973"},{"key":"e_1_3_2_169_2","first-page":"4514","article-title":"Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training","volume":"34","author":"Xue Hongwei","year":"2021","unstructured":"Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. 2021. Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training. Advances in Neural Information Processing Systems 34 (2021), 4514\u20134528.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_170_2","article-title":"Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V","author":"Yang Jianwei","year":"2023","unstructured":"Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4V. arXiv preprint arXiv:2310.11441 (2023).","journal-title":"arXiv preprint arXiv:2310.11441"},{"key":"e_1_3_2_171_2","article-title":"Gpt4tools: Teaching large language model to use tools via self-instruction","volume":"36","author":"Yang Rui","year":"2024","unstructured":"Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2024. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_172_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00220"},{"key":"e_1_3_2_173_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i3.20215"},{"key":"e_1_3_2_174_2","first-page":"3081","volume-title":"Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22\u2013March 1, 2022","author":"Yang Zhengyuan","year":"2022","unstructured":"Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of GPT-3 for few-shot knowledge-based VQA. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22\u2013March 1, 2022. AAAI Press, 3081\u20133089."},{"key":"e_1_3_2_175_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.10"},{"key":"e_1_3_2_176_2","article-title":"MM-REACT: Prompting ChatGPT for multimodal reasoning and action","volume":"2303","author":"Yang Zhengyuan","year":"2023","unstructured":"Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. MM-REACT: Prompting ChatGPT for multimodal reasoning and action. CoRR abs\/2303.11381 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_177_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2023.3267036"},{"key":"e_1_3_2_178_2","article-title":"mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration","volume":"2311","author":"Ye Qinghao","year":"2023","unstructured":"Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration. CoRR abs\/2311.04257 (2023). arXiv:2311.04257","journal-title":"CoRR"},{"key":"e_1_3_2_179_2","article-title":"Idealgpt: Iteratively decomposing vision and language reasoning via large language models","author":"You Haoxuan","year":"2023","unstructured":"Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A. Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. 2023. Idealgpt: Iteratively decomposing vision and language reasoning via large language models. arXiv preprint arXiv:2305.14985 (2023).","journal-title":"arXiv preprint arXiv:2305.14985"},{"key":"e_1_3_2_180_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i4.16431"},{"key":"e_1_3_2_181_2","article-title":"Visual madlibs: Fill in the blank image generation and question answering","author":"Yu Licheng","year":"2015","unstructured":"Licheng Yu, Eunbyung Park, Alexander C. Berg, and Tamara L. Berg. 2015. Visual madlibs: Fill in the blank image generation and question answering. arXiv preprint arXiv:1506.00278 (2015).","journal-title":"arXiv preprint arXiv:1506.00278"},{"key":"e_1_3_2_182_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.202"},{"key":"e_1_3_2_183_2","volume-title":"Proceedings of CVPR","author":"Yue Xiang","year":"2024","unstructured":"Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.. 2024. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of CVPR."},{"key":"e_1_3_2_184_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2024.3456159"},{"key":"e_1_3_2_185_2","first-page":"11941","volume-title":"IEEE\/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1\u20136, 2023","author":"Zhai Xiaohua","year":"2023","unstructured":"Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In IEEE\/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1\u20136, 2023. IEEE, 11941\u201311952."},{"key":"e_1_3_2_186_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.03.005"},{"key":"e_1_3_2_187_2","article-title":"CMMMU: A chinese massive multi-discipline multimodal understanding benchmark","author":"Zhang Ge","year":"2024","unstructured":"Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et\u00a0al. 2024. CMMMU: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944 (2024).","journal-title":"arXiv preprint arXiv:2401.11944"},{"key":"e_1_3_2_188_2","volume-title":"NAACL","author":"Zhang Jiawei","year":"2024","unstructured":"Jiawei Zhang, Tianyu Pang, Chao Du, Yi Ren, Bo Li, and Min Lin. 2024. Benchmarking large multimodal models against common corruptions. In NAACL."},{"key":"e_1_3_2_189_2","article-title":"Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?","author":"Zhang Renrui","year":"2024","unstructured":"Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et\u00a0al. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624 (2024).","journal-title":"arXiv preprint arXiv:2403.14624"},{"key":"e_1_3_2_190_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3612837"},{"key":"e_1_3_2_191_2","article-title":"Multimodal chain-of-thought reasoning in language models","volume":"2302","author":"Zhang Zhuosheng","year":"2023","unstructured":"Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. CoRR abs\/2302.00923 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_192_2","article-title":"MMICL: Empowering vision-language model with multi-modal in-context learning","volume":"2309","author":"Zhao Haozhe","year":"2023","unstructured":"Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2023. MMICL: Empowering vision-language model with multi-modal in-context learning. CoRR abs\/2309.07915 (2023). arXiv:2309.07915","journal-title":"CoRR"},{"key":"e_1_3_2_193_2","article-title":"Minigpt-4: Enhancing vision-language understanding with advanced large language models","author":"Zhu Deyao","year":"2023","unstructured":"Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023).","journal-title":"arXiv preprint arXiv:2304.10592"},{"key":"e_1_3_2_194_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.540"},{"key":"e_1_3_2_195_2","series-title":"IJCAI\u201920","volume-title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","author":"Zhu Zihao","year":"2021","unstructured":"Zihao Zhu, Jing Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu. 2021. Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (Yokohama, Yokohama, Japan) (IJCAI\u201920). Article 153, 7 pages."},{"key":"e_1_3_2_196_2","doi-asserted-by":"crossref","first-page":"1863","DOI":"10.18653\/v1\/2020.coling-main.169","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Ziaeefard Maryam","year":"2020","unstructured":"Maryam Ziaeefard and Freddy Lecue. 2020. Towards knowledge-augmented visual question answering. In Proceedings of the 28th International Conference on Computational Linguistics. 1863\u20131873."},{"key":"e_1_3_2_197_2","first-page":"289","volume-title":"2020 2nd International Conference on Information Technology and Computer Application (ITCA\u201920)","author":"Zou Yeyun","year":"2020","unstructured":"Yeyun Zou and Qiyu Xie. 2020. A survey on VQA: Datasets and approaches. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA\u201920). IEEE, 289\u2013297."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711680","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3711680","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:19:15Z","timestamp":1750295955000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711680"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,5]]},"references-count":196,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2025,8,31]]}},"alternative-id":["10.1145\/3711680"],"URL":"https:\/\/doi.org\/10.1145\/3711680","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,5]]},"assertion":[{"value":"2023-01-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}