{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T03:38:03Z","timestamp":1782358683083,"version":"3.54.5"},"reference-count":149,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,12,10]],"date-time":"2024-12-10T00:00:00Z","timestamp":1733788800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,10]],"date-time":"2024-12-10T00:00:00Z","timestamp":1733788800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2022ZD0161300"],"award-info":[{"award-number":["2022ZD0161300"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62376134"],"award-info":[{"award-number":["62376134"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62372223"],"award-info":[{"award-number":["62372223"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Vis. Intell."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Multi-modal large language models (MLLMs) have demonstrated impressive performance in vision-language tasks across a wide range of domains. However, the large model scale and associated high computational cost pose significant challenges for training and deploying MLLMs on consumer-grade GPUs or edge devices, thereby hindering their widespread application. In this work, we introduce Mini-InternVL, a series of MLLMs with parameters ranging from 1 billion to 4 billion, which achieves 90% of the performance with only 5% of the parameters. This significant improvement in efficiency and effectiveness makes our models more accessible and applicable in various real-world scenarios. To further promote the adoption of our models, we are developing a unified adaptation framework for Mini-InternVL, which enables our models to transfer and outperform specialized models in downstream tasks, including autonomous driving, medical image processing, and remote sensing. We believe that our models can provide valuable insights and resources to advance the development of efficient and effective MLLMs.<\/jats:p>","DOI":"10.1007\/s44267-024-00067-6","type":"journal-article","created":{"date-parts":[[2024,12,10]],"date-time":"2024-12-10T09:56:20Z","timestamp":1733824580000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance"],"prefix":"10.1007","volume":"2","author":[{"given":"Zhangwei","family":"Gao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhe","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erfei","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yiming","family":"Ren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiyun","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinguo","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Tian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shenglong","family":"Ye","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junjun","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xizhou","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lewei","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tong","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu","family":"Qiao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jifeng","family":"Dai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2418-3134","authenticated-orcid":false,"given":"Wenhai","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,12,10]]},"reference":[{"key":"67_CR1","first-page":"24185","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Z. Chen","year":"2024","unstructured":"Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., et al. (2024). InternVL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 24185\u201324198). Piscataway: IEEE."},{"key":"67_CR2","unstructured":"Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., et\u00a0al. (2024). How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites. arXiv preprint. arXiv:2404.16821."},{"key":"67_CR3","unstructured":"OpenAI. GPT-4v(ision) system card. (2023). Retrieved November 20, 2024, from https:\/\/cdn.openai.com\/papers\/GPTV_System_Card.pdf."},{"key":"67_CR4","unstructured":"Hello, O. (2024). GPT-4o. Retrieved November 20, 2024, from https:\/\/openai.com\/index\/hello-gpt-4o\/."},{"key":"67_CR5","unstructured":"Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., et\u00a0al. (2024). LLaVA-OneVision: easy visual task transfer. arXiv preprint. arXiv:2408.03326."},{"key":"67_CR6","unstructured":"Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., et\u00a0al. (2024). LLaVA-next: improved reasoning, OCR, and world knowledge. Retrieved November 20, 2024, from https:\/\/llava-vl.github.io\/blog\/2024-01-30-llava-next\/."},{"key":"67_CR7","unstructured":"Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., et\u00a0al. (2024). Emu3: next-token prediction is all you need. arXiv preprint. arXiv:2409.18869."},{"key":"67_CR8","first-page":"26689","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"J. Lin","year":"2024","unstructured":"Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Vila, S. H. (2024). On pre-training for visual language models. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 26689\u201326699). Piscataway: IEEE."},{"key":"67_CR9","unstructured":"Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., et\u00a0al. (2024). mPLUG-Owl3: towards long image-sequence understanding in multi-modal large language models. arXiv preprint. arXiv:2408.04840."},{"key":"67_CR10","unstructured":"Ge, C., Cheng, S., Wang, Z., Yuan, J., Gao, Y., Song, J., et\u00a0al. (2024). ConvLLaVA: hierarchical backbones as visual encoder for large multimodal models. arXiv preprint. arXiv:2405.15738."},{"key":"67_CR11","unstructured":"Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., et\u00a0al. (2024). Eagle: exploring the design space for multimodal LLMs with mixture of encoders. arXiv preprint. arXiv:2408.15998."},{"key":"67_CR12","unstructured":"Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., et\u00a0al. (2024). The Llama 3 herd of models. arXiv preprint. arXiv:2407.21783."},{"key":"67_CR13","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et\u00a0al. (2023). Llama 2: open foundation and fine-tuned chat models. arXiv preprint. arXiv:2307.09288."},{"key":"67_CR14","unstructured":"Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., et\u00a0al. (2024). InternLM2 technical report. arXiv preprint. arXiv:2403.17297."},{"key":"67_CR15","unstructured":"Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., et\u00a0al. (2023). Qwen technical report. arXiv preprint. arXiv:2309.16609."},{"key":"67_CR16","unstructured":"Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., et\u00a0al. (2024). Qwen2 technical report. arXiv preprint. arXiv:2407.10671."},{"key":"67_CR17","unstructured":"Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., & de las Casas, D., et\u00a0al. (2023). Mistral 7b. arXiv preprint. arXiv:2310.06825."},{"key":"67_CR18","first-page":"8748","volume-title":"Proceedings of the 38th international conference on machine learning","author":"A. Radford","year":"2021","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., et al. (2021). Learning transferable visual models from natural language supervision. In M. Meila & T. Zhang (Eds.), Proceedings of the 38th international conference on machine learning. (pp. 8748\u20138763). Retrieved November 20, 2024, from http:\/\/proceedings.mlr.press\/v139\/radford21a.html."},{"key":"67_CR19","first-page":"11975","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"X. Zhai","year":"2023","unstructured":"Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid loss for language image pre-training. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 11975\u201311986). Piscataway: IEEE."},{"key":"67_CR20","unstructured":"Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., et\u00a0al. (2024). MiniCPM-V: a GPT-4V level MLLM on your phone. arXiv preprint. arXiv:2408.01800."},{"key":"67_CR21","unstructured":"Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., et\u00a0al. (2023). Mini-Gemini: mining the potential of multi-modality vision language models. arXiv preprint. arXiv:2403.18814."},{"key":"67_CR22","unstructured":"Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., et\u00a0al. (2023). MobileVLM: a fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint. arXiv:2312.16886."},{"key":"67_CR23","unstructured":"Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., et\u00a0al. (2024). MobileVLM v2: faster and stronger baseline for vision language model. arXiv preprint. arXiv:2402.03766."},{"key":"67_CR24","unstructured":"Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., et\u00a0al. (2024). PaliGemma: a versatile 3B VLM for transfer. arXiv preprint. arXiv:2407.07726."},{"key":"67_CR25","unstructured":"He, M., Liu, Y., Wu, B., Yuan, J., Wang, Y., Huang, T., et\u00a0al. (2024). Efficient multimodal learning from data-centric perspective. arXiv preprint. arXiv:2402.11530."},{"key":"67_CR26","first-page":"4171","volume-title":"Proceedings of the 2019 conference of the North American Chapter of the Association for computational linguistics: human language technologies","author":"J. Devlin","year":"2019","unstructured":"Devlin, J. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the North American Chapter of the Association for computational linguistics: human language technologies (pp. 4171\u20134186). Stroudsburg: ACL."},{"key":"67_CR27","first-page":"4904","volume-title":"Proceedings of the 38th international conference on machine learning","author":"C. Jia","year":"2021","unstructured":"Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., et al. (2021). Scaling up visual and vision-language representation learning with noisy text supervision. In M. Meila & T. Zhang (Eds.), Proceedings of the 38th international conference on machine learning (pp. 4904\u20134916). Retrieved November 20, 2024, from http:\/\/proceedings.mlr.press\/v139\/jia21b.html."},{"key":"67_CR28","unstructured":"Abdin, M., Jacobs, S.A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., et\u00a0al. (2024). Phi-3 technical report: a highly capable language model locally on your phone. arXiv preprint. arXiv:2404.14219."},{"key":"67_CR29","first-page":"216","volume-title":"Proceedings of the 18th European conference on computer vision workshops","author":"Y. Liu","year":"2024","unstructured":"Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., et al. (2024). MMBench: is your multi-modal model an all-around player? In A. Leonardis, E. Ricci, S. Roth, et al. (Eds.), Proceedings of the 18th European conference on computer vision workshops (pp. 216\u2013233). Cham: Springer."},{"key":"67_CR30","first-page":"2263","volume-title":"Findings of the Association for Computational Linguistics","author":"A. Masry","year":"2022","unstructured":"Masry, A., Do, X.L., Qing, J.T., Joty, S., & Hoque, E. (2022). ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), Findings of the Association for Computational Linguistics (pp. 2263\u20132279). Stroudsburg: ACL."},{"key":"67_CR31","first-page":"1","volume-title":"Proceedings of the 12th international conference on learning representations","author":"P. Lu","year":"2024","unstructured":"Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., et al. (2024). MathVista: evaluating mathematical reasoning of foundation models in visual contexts. In Proceedings of the 12th international conference on learning representations (pp. 1\u2013116). Retrieved November 20, 2024, from https:\/\/openreview.net\/forum?id=KUNzEQMWU7."},{"key":"67_CR32","unstructured":"Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., et\u00a0al. (2023). InternGPT: solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint. arXiv:2305.05662."},{"key":"67_CR33","first-page":"1","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"Y. Shen","year":"2023","unstructured":"Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023). HuggingGPT: solving ai tasks with ChatGPT and its friends in hugging face. In A. Oh, T. Neumann, A. Globerson, et al. (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp. 1\u201327). Red Hook: Curran Associates."},{"key":"67_CR34","unstructured":"Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023). Visual ChatGPT: talking, drawing and editing with visual foundation models. arXiv preprint. arXiv:2303.04671."},{"key":"67_CR35","first-page":"12888","volume-title":"Proceedings of the 39th international conference on machine learning","author":"J. Li","year":"2022","unstructured":"Li, J., Li, D., Xiong, C., & Hoi, S. (2022). BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In K. Chaudhuri, S. Jegelka, L. Song, et al. (Eds.), Proceedings of the 39th international conference on machine learning (pp. 12888\u201312900). Retrieved November 20, 2024, from https:\/\/proceedings.mlr.press\/v162\/li22n\/li22n.pdf."},{"key":"67_CR36","first-page":"19730","volume-title":"Proceedings of the 40th international conference on machine learning","author":"J. Li","year":"2023","unstructured":"Li, J., Li, D., Savarese, S., & Hoi, S. (2023). BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th international conference on machine learning (pp. 19730\u201319742). Retrieved November 20, 2024, from https:\/\/proceedings.mlr.press\/v202\/li23q\/li23q.pdf."},{"key":"67_CR37","first-page":"1","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"H. Liu","year":"2023","unstructured":"Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning. In A. Oh, T. Neumann, A. Globerson, et al. (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp. 1\u201325). Red Hook: Curran Associates."},{"key":"67_CR38","unstructured":"Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., et\u00a0al. (2023). Qwen-VL: a versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint. arXiv:2308.12966."},{"key":"67_CR39","unstructured":"Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., et\u00a0al. (2023). Kosmos-2: grounding multimodal large language models to the world. arXiv preprint. arXiv:2306.14824."},{"key":"67_CR40","unstructured":"Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., et\u00a0al. (2024). Needle in a multimodal haystack. arXiv preprint. arXiv:2406.07230."},{"key":"67_CR41","first-page":"23716","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"J.-B. Alayrac","year":"2022","unstructured":"Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., et al. (2022). Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, et al. (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp. 23716\u201323736). Red Hook: Curran Associates."},{"key":"67_CR42","unstructured":"Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., et\u00a0al. (2023). CogVLM: visual expert for pretrained language models. arXiv preprint. arXiv:2311.03079."},{"key":"67_CR43","unstructured":"Tian, C., Zhu, X., Xiong, Y., Wang, W., Chen, Z., Wang, W., et\u00a0al. (2024). MM-interleaved: interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint. arXiv:2401.10208."},{"key":"67_CR44","unstructured":"Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., et\u00a0al. (2023). Introducing our multimodal models. Retrieved November 20, 2024, from https:\/\/www.adept.ai\/blog\/fuyu-8b."},{"key":"67_CR45","unstructured":"Lin, X.V., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Gosh, G., et al. (2024). Efficient early-fusion pre-training with mixture of modality-aware experts. arXiv preprint. arXiv:2407.21770."},{"key":"67_CR46","unstructured":"Chameleon team (2024). Chameleon: mixed-modal early-fusion foundation models. arXiv preprint. arXiv:2405.09818."},{"key":"67_CR47","unstructured":"Sun, Q., Fang, Y., Wu, L., Wang, X., & Cao, Y. (2023). EVA-CLIP: improved training techniques for clip at scale. arXiv preprint. arXiv:2303.15389."},{"key":"67_CR48","unstructured":"Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., et\u00a0al. (2024). DeepSeek-VL: towards real-world vision-language understanding. arXiv preprint. arXiv:2403.05525."},{"key":"67_CR49","first-page":"1","volume-title":"Proceedings of the 12th international conference on learning representations","author":"W. Wang","year":"2024","unstructured":"Wang, W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, L., et al. (2024). The All-Seeing project: towards panoptic visual recognition and understanding of the open world. In Proceedings of the 12th international conference on learning representations (pp. 1\u201333). Retrieved November 20, 2024, from https:\/\/openreview.net\/forum?id=c2R7ajodcI."},{"key":"67_CR50","first-page":"471","volume-title":"Proceedings of the 18th European conference on computer vision","author":"W. Wang","year":"2024","unstructured":"Wang, W., Ren, Y., Luo, H., Li, T., Yan, C., Chen, Z., et al. (2024). The All-Seeing project v2: towards general relation comprehension of the open world. In A. Leonardis, E. Ricci & S. Roth (Eds.), Proceedings of the 18th European conference on computer vision (pp. 471\u2013490). Cham: Springer."},{"key":"67_CR51","first-page":"9568","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"S. Tong","year":"2024","unstructured":"Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024). Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 9568\u20139578). Piscataway: IEEE."},{"key":"67_CR52","unstructured":"Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M. Khalidov, V., et\u00a0al. (2023). DINOv2: learning robust visual features without supervision. arXiv preprint. arXiv:2304.07193."},{"key":"67_CR53","unstructured":"Luo, G., Zhou, Y., Zhang, Y., Zheng, X., Sun, X., & Ji, R. (2024). Feast your eyes: mixture-of-resolution adaptation for multimodal large language models. arXiv preprint. arXiv:2403.03003."},{"key":"67_CR54","first-page":"3992","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"A. Kirillov","year":"2023","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., et al. (2023). Segment anything. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 3992\u20134003). Piscataway: IEEE."},{"key":"67_CR55","first-page":"27831","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"K. Kuckreja","year":"2024","unstructured":"Kuckreja, K., Danish, M. S., Naseer, M., Das, A., Khan, S., & Khan, F. S. (2024). GeoChat: grounded large vision-language model for remote sensing. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 27831\u201327840). Piscataway: IEEE."},{"key":"67_CR56","doi-asserted-by":"crossref","unstructured":"Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X. (2024). EarthGPT: a universal multi-modal large language model for multi-sensor image comprehension in remote sensing domain. arXiv preprint. arXiv:2401.16822.","DOI":"10.1109\/TGRS.2024.3409624"},{"key":"67_CR57","first-page":"8406","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"C. Li","year":"2023","unstructured":"Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., et al. (2023). LLaVA-Med: training a large language-and-vision assistant for biomedicine in one day. In A. Oh, T. Neumann, A. Globerson, et al. (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp. 8406\u20138441). Red Hook: Curran Associates."},{"key":"67_CR58","unstructured":"Ji, G.-P., Liu, J., Xu, P., Barnes, N., Khan, F.S., Khan, S., et\u00a0al. (2024). Frontiers in intelligent colonoscopy. arXiv preprint. arXiv:2410.17241."},{"key":"67_CR59","unstructured":"Liu, J., Wang, Z., Ye, Q., Chong, D., Zhou, P., & Hua, Y. (2023). Qilin-Med-VL: towards Chinese large vision-language model for general healthcare. arXiv preprint. arXiv:2310.17956."},{"key":"67_CR60","unstructured":"Li, J., Zhang, D., Wang, X., Hao, Z., Lei, J., Tan, Q., et\u00a0al. (2024). Seeing and understanding: bridging vision with chemical knowledge via ChemVLM. arXiv preprint. arXiv:2408.07246."},{"key":"67_CR61","unstructured":"Tian, X., Gu, J., Li, B., Liu, Y., Hu, C., Wang, Y., et\u00a0al. (2024). DriveVLM: the convergence of autonomous driving and large vision-language models. arXiv preprint. arXiv:2402.12289."},{"key":"67_CR62","unstructured":"Wang, W., Xie, J., Hu, C., Zou, H., Fan, J., Tong, W., et\u00a0al. (2023). DriveMLM: aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint. arXiv:2312.09245."},{"issue":"10","key":"67_CR63","doi-asserted-by":"publisher","first-page":"8186","DOI":"10.1109\/LRA.2024.3440097","volume":"9","author":"Z. Xu","year":"2024","unstructured":"Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.-Y. K., et al. (2024). DriveGPT4: interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 9(10), 8186\u20138193.","journal-title":"IEEE Robotics and Automation Letters"},{"key":"67_CR64","first-page":"25278","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"C. Schuhmann","year":"2022","unstructured":"Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., et al. (2022). LAION-5B: an open large-scale dataset for training next generation image-text models. In S. Koyejo, S. Mohamed, A. Agarwal, et al. (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp. 25278\u201325294). Red Hook: Curran Associates."},{"key":"67_CR65","unstructured":"Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., & Kim, S. (2022). COYO-700m: image-text pair dataset. Retrieved November 20, 2024, from https:\/\/github.com\/kakaobrain\/coyo-dataset."},{"key":"67_CR66","unstructured":"Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Doll\u00e1r, P., et\u00a0al. (2015). Microsoft COCO captions: data collection and evaluation server. arXiv preprint. arXiv:1504.00325."},{"key":"67_CR67","first-page":"5356","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"A. Gupta","year":"2019","unstructured":"Gupta, A., Dollar, P., & Girshick, R. (2019). LVIS: a dataset for large vocabulary instance segmentation. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 5356\u20135364). Piscataway: IEEE."},{"key":"67_CR68","first-page":"8430","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"S. Shao","year":"2019","unstructured":"Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., et al. (2019). Objects365: a large-scale, high-quality dataset for object detection. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 8430\u20138439). Piscataway: IEEE."},{"key":"67_CR69","first-page":"2641","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"B. A. Plummer","year":"2015","unstructured":"Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015). Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 2641\u20132649). Piscataway: IEEE."},{"issue":"1","key":"67_CR70","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","volume":"123","author":"R. Krishna","year":"2017","unstructured":"Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., et al. (2017). Visual genome: connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1), 32\u201373.","journal-title":"International Journal of Computer Vision"},{"key":"67_CR71","unstructured":"Liu, Y., Cao, Y., Gao, Z., Wang, W., Chen, Z., Wang, W., et\u00a0al. (2024). MMInstruct: a high-quality multi-modal instruction tuning dataset with extensive diversity. arXiv preprint. arXiv:2407.15838."},{"key":"67_CR72","unstructured":"Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023). Aligning large multi-modal model with robust instruction tuning. arXiv preprint. arXiv:2306.14565."},{"key":"67_CR73","first-page":"742","volume-title":"Proceedings of the 16th European conference on computer vision","author":"O. Sidorov","year":"2020","unstructured":"Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020). TextCaps: a dataset for image captioning with reading comprehension. In A. Vedaldi, H. Bischof, T. Brox, et al. (Eds.), Proceedings of the 16th European conference on computer vision (pp. 742\u2013758). Cham: Springer."},{"key":"67_CR74","first-page":"26418","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"J. Gu","year":"2022","unstructured":"Gu, J., Meng, X., Lu, G., Hou, L., Minzhe, N., Liang, X., et al. (2022). Wukong: a 100 million large-scale Chinese cross-modal pre-training benchmark. In S. Koyejo, S. Mohamed, A. Agarwal, et al. (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp. 26418\u201326431). Red Hook: Curran Associates."},{"issue":"3","key":"67_CR75","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1007\/s11390-019-1923-y","volume":"34","author":"T.-L. Yuan","year":"2019","unstructured":"Yuan, T.-L., Zhu, Z., Xu, K., Li, C.-J., Mu, T.-J., & Hu, S.-M. (2019). A large Chinese text dataset in the wild. Journal of Computer Science and Technology, 34(3), 509\u2013521.","journal-title":"Journal of Computer Science and Technology"},{"key":"67_CR76","first-page":"1287","volume-title":"Proceedings of the 2019 conference of the North American Chapter of the Association for computational linguistics: human language technologies","author":"F. Liu","year":"2024","unstructured":"Liu, F., Wang, X., Yao, W., Chen, J., Song, K., Cho, S., et al. (2024). MMC: advancing multimodal chart understanding with large-scale instruction tuning. In K. Duh, H. G\u00f3mez-Adorno, & S. Bethard (Eds.), Proceedings of the 2019 conference of the North American Chapter of the Association for computational linguistics: human language technologies (pp. 1287\u20131310). Stroudsburg: ACL."},{"key":"67_CR77","first-page":"1557","volume-title":"Proceedings of the 2019 international conference on document analysis and recognition","author":"Y. Sun","year":"2019","unstructured":"Sun, Y., Ni, Z., Chng, C.-K., Liu, Y., Luo, C., Ng, C.C., et al. (2019). ICDAR 2019 competition on large-scale street view text with partial labeling-RRC-LSVT. In Proceedings of the 2019 international conference on document analysis and recognition (pp. 1557\u20131562). Piscataway: IEEE."},{"key":"67_CR78","first-page":"4291","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"A.F. Biten","year":"2019","unstructured":"Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., et al. (2019). Scene text visual question answering. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 4291\u20134301). Piscataway: IEEE."},{"key":"67_CR79","first-page":"1429","volume-title":"Proceedings of the 2017 international conference on document analysis and recognition","author":"B. Shi","year":"2017","unstructured":"Shi, B., Yao, C., Liao, M., Yang, M., Xu, P., Cui, L., et al. (2017). ICDAR 2017 competition on reading Chinese text in the wild (RCTW-17). In Proceedings of the 2017 international conference on document analysis and recognition (pp. 1429\u20131434). Piscataway: IEEE."},{"key":"67_CR80","first-page":"1577","volume-title":"Proceedings of the 2019 international conference on document analysis and recognition","author":"R. Zhang","year":"2019","unstructured":"Zhang, R., Zhou, Y., Jiang, Q., Song, Q., Li, N., Zhou, K., et al. (2019). ICDAR 2019 robust reading challenge on reading Chinese text on signboard. In Proceedings of the 2019 international conference on document analysis and recognition (pp. 1577\u20131581). Piscataway: IEEE."},{"key":"67_CR81","first-page":"1571","volume-title":"Proceedings of the 2019 international conference on document analysis and recognition","author":"C.K. Chng","year":"2019","unstructured":"Chng, C.K., Ding, E., Liu, J., Karatzas, D., Chan, C. S., Lin, L., et al. (2019). ICDAR 2019 robust reading challenge on arbitrary-shaped text-RRC-ArT. In Proceedings of the 2019 international conference on document analysis and recognition (pp. 1571\u20131576). Piscataway: IEEE."},{"key":"67_CR82","first-page":"498","volume-title":"Proceedings of the 17th European conference on computer vision workshops","author":"G. Kim","year":"2022","unstructured":"Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., et al. (2022). OCR-free document understanding transformer. In S. Avidan, G. J. Brostow, M. Ciss\u00e9, et al. (Eds.), Proceedings of the 17th European conference on computer vision workshops (pp. 498\u2013517). Cham: Springer."},{"key":"67_CR83","unstructured":"Schuhmann, C., K\u00f6pf, A., Vencu, R., Coombes, T., & Beaumont, R. (2024). Laion COCO: 600m synthetic captions from laion2b-en. Retrieved November 20, 2024, from https:\/\/laion.ai\/blog\/laion-coco\/."},{"key":"67_CR84","unstructured":"Veit, A., Matera, T., Neumann, L., Matas, J., & Belongie, S. (2016). COCO-text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint. arXiv:1601.07140."},{"key":"67_CR85","first-page":"2200","volume-title":"Proceedings of IEEE winter conference on applications of computer vision","author":"M. Mathew","year":"2021","unstructured":"Mathew, M., Karatzas, D., & Jawahar, C. V. (2021). DocVQA: a dataset for VQA on document images. In Proceedings of IEEE winter conference on applications of computer vision (pp. 2200\u20132209). Piscataway: IEEE."},{"key":"67_CR86","first-page":"8802","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"A. Singh","year":"2021","unstructured":"Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., & Hassner, T. (2021). TextOCR: towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 8802\u20138812). Piscataway: IEEE."},{"key":"67_CR87","unstructured":"Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., et\u00a0al. (2023). LLaVAR: enhanced visual instruction tuning for text-rich image understanding. arXiv preprint. arXiv:2306.17107."},{"key":"67_CR88","first-page":"4999","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"A. Kembhavi","year":"2017","unstructured":"Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017). Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 4999\u20135007). Piscataway: IEEE."},{"key":"67_CR89","first-page":"2315","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"A. Gupta","year":"2016","unstructured":"Gupta, A., Vedaldi, A., & Zisserman, A. (2016). Synthetic data for text localisation in natural images. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 2315\u20132324). Piscataway: IEEE."},{"key":"67_CR90","first-page":"3096","volume-title":"Findings of the Association for Computational Linguistics","author":"A. Hu","year":"2024","unstructured":"Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., et al. (2024). mPLUG-DOCOwl 1.5: unified structure learning for OCR-free document understanding. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Findings of the Association for Computational Linguistics (pp. 3096\u20133120). Stroudsburg: ACL."},{"key":"67_CR91","first-page":"235","volume-title":"Proceedings of the 14th European conference on computer vision workshops","author":"A. Kembhavi","year":"2016","unstructured":"Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., & Farhadi, A. (2016). A diagram is worth a dozen images. In B. Leibe, J. Matas, N. Sebe, et al. (Eds.), Proceedings of the 14th European conference on computer vision workshops (pp. 235\u2013251). Cham: Springer."},{"key":"67_CR92","first-page":"1527","volume-title":"Proceedings of IEEE winter conference on applications of computer vision","author":"N. Methani","year":"2020","unstructured":"Methani, N., Ganguly, P., Khapra, M. M., & Kumar, P. (2020). PlotQA: reasoning over scientific plots. In Proceedings of IEEE winter conference on applications of computer vision (pp. 1527\u20131536). Piscataway: IEEE."},{"key":"67_CR93","first-page":"1697","volume-title":"Proceedings of IEEE winter conference on applications of computer vision","author":"M. Mathew","year":"2022","unstructured":"Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., & Jawahar, C. (2022). InfographicVQA. In Proceedings of IEEE winter conference on applications of computer vision (pp. 1697\u20131706). Piscataway: IEEE."},{"key":"67_CR94","first-page":"1","volume-title":"Proceedings of the 36th international conference on neural information processing systemsworkshops","author":"S. Chang","year":"2022","unstructured":"Chang, S., Palzer, D., Li, J., Fosler-Lussier, E., & Xiao, N. (2022). MapQA: a dataset for question answering on choropleth maps. In Proceedings of the 36th international conference on neural information processing systemsworkshops (pp. 1\u201313). Red Hook: Curran Associates."},{"key":"67_CR95","first-page":"1","volume-title":"Proceedings of the 6th international conference on learning representations","author":"S.E. Kahou","year":"2018","unstructured":"Kahou, S.E., Michalski, V., Atkinson, A., K\u00e1d\u00e1r, \u00c1., Trischler, A., & Bengio, Y. (2018). FigureQA: an annotated figure dataset for visual reasoning. In Proceedings of the 6th international conference on learning representations (pp. 1\u201320). Retrieved November 20, 2024, from https:\/\/openreview.net\/forum?id=SyunbfbAb."},{"key":"67_CR96","first-page":"1","volume-title":"Proceedings of the neural information processing systems track on datasets and benchmarks 1","author":"P. Lu","year":"2021","unstructured":"Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., et al. (2021). IconQA: a new benchmark for abstract diagram understanding and visual language reasoning. In J. Vanschoren & S.-K. Yeung (Eds.), Proceedings of the neural information processing systems track on datasets and benchmarks 1 (pp. 1\u201314). Red Hook: Curran Associates."},{"key":"67_CR97","first-page":"14963","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Z. Li","year":"2023","unstructured":"Li, Z., Wang, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., van Durme, B., et al. (2023). Super-CLEVR: a virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 14963\u201314973). Piscataway: IEEE."},{"key":"67_CR98","first-page":"155","volume-title":"Proceedings of the 16th international workshop on neural-symbolic learning and reasoning","author":"A.D. Lindstr\u00f6m","year":"2022","unstructured":"Lindstr\u00f6m, A.D., & Abraham, S. S. (2022). CLEVR-Math: a dataset for compositional language, visual and mathematical reasoning. In A. S. d\u2019Avila Garcez & E. Jimenez-Ruiz (Eds.), Proceedings of the 16th international workshop on neural-symbolic learning and reasoning (pp. 155\u2013170). Red Hook: Curran Associates."},{"key":"67_CR99","first-page":"1511","volume-title":"Proceedings of the 29th international conference on computational linguistics","author":"J. Cao","year":"2022","unstructured":"Cao, J., & Xiao, J. (2022). An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In N. Calzolari, C.-R. Huang, H. Kim, et al. (Eds.), Proceedings of the 29th international conference on computational linguistics (pp. 1511\u20131520). Red Hook: Curran Associates."},{"key":"67_CR100","doi-asserted-by":"publisher","first-page":"14662","DOI":"10.18653\/v1\/2023.emnlp-main.906","volume-title":"Proceedings of the 2023 conference on empirical methods in natural language processing","author":"A. Masry","year":"2023","unstructured":"Masry, A., Kavehzadeh, P., Do, X.L., Hoque, E., & Joty, S. (2023). UniChart: a universal vision-language pretrained model for chart comprehension and reasoning. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 14662\u201314684). Stroudsbrug: ACL."},{"key":"67_CR101","first-page":"2507","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"P. Lu","year":"2022","unstructured":"Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., et al. (2022). Learn to explain: multimodal reasoning via thought chains for science question answering. In Proceedings of the 36th international conference on neural information processing systems (pp. 2507\u20132521). Red Hook: Curran Associates."},{"key":"67_CR102","first-page":"6774","volume-title":"Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing","author":"P. Lu","year":"2021","unstructured":"Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., et al. (2021). Inter-GPS: interpretable geometry problem solving with formal language and symbolic reasoning. In C. Zong, F. Xia, W. Li, et al. (Eds.), Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing (pp. 6774\u20136786). Stroudsburg: ACL."},{"key":"67_CR103","doi-asserted-by":"publisher","first-page":"3313","DOI":"10.18653\/v1\/2022.emnlp-main.218","volume-title":"Proceedings of the 2022 conference on empirical methods in natural language processing","author":"J. Chen","year":"2022","unstructured":"Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., et al. (2022). UniGeo: unifying geometry logical reasoning via reformulating mathematical expression. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 conference on empirical methods in natural language processing (pp. 3313\u20133323). Stroudsbrug: ACL."},{"key":"67_CR104","unstructured":"Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., et\u00a0al. (2023). PMC-VQA: visual instruction tuning for medical visual question answering. arXiv preprint. arXiv:2305.10415."},{"key":"67_CR105","first-page":"1","volume-title":"Proceedings of the 11th international conference on learning representations","author":"P. Lu","year":"2023","unstructured":"Lu, P., Qiu, L., Chang, K.-W., Wu, Y.N., Zhu, S.-C., Rajpurohit, T., et al. (2023). Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In Proceedings of the 11th international conference on learning representations (pp. 1\u201326). Retrieved November 20, 2024, from https:\/\/openreview.net\/forum?id=ceFkn2lhOv."},{"key":"67_CR106","first-page":"1","volume-title":"Proceedings of the 12th international conference on learning representations","author":"L. Yu","year":"2024","unstructured":"Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., et al. (2024). MetaMath: bootstrap your own mathematical questions for large language models. In Proceedings of the 12th international conference on learning representations (pp. 1\u201322). Retrieved November 20, 2024, from https:\/\/openreview.net\/forum?id=N8N0hgNDRt."},{"key":"67_CR107","first-page":"1331","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"B. Yao","year":"2011","unstructured":"Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., & Li, F.-F. (2011). Human action recognition by learning bases of action attributes and parts. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 1331\u20131338). Piscataway: IEEE."},{"key":"67_CR108","first-page":"6700","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"D. A. Hudson","year":"2019","unstructured":"Hudson, D. A., & Manning, C. D. (2019). GQA: a new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 6700\u20136709). Piscataway: IEEE."},{"key":"67_CR109","first-page":"709","volume-title":"Proceedings of the 16th European conference on computer vision","author":"Q. Huang","year":"2020","unstructured":"Huang, Q., Xiong, Y., Rao, A., Wang, J., & Lin, D. (2020). MovieNet: a holistic dataset for movie understanding. In A. Vedaldi, H. Bischof, T. Brox, et al. (Eds.), Proceedings of the 16th European conference on computer vision (pp. 709\u2013727). Cham: Springer."},{"key":"67_CR110","doi-asserted-by":"publisher","first-page":"4041","DOI":"10.1109\/TIP.2020.2967829","volume":"29","author":"V. Hosu","year":"2020","unstructured":"Hosu, V., Lin, H., Sziranyi, T., & Saupe, D. (2020). KonIQ-10k: an ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 29, 4041\u20134056.","journal-title":"IEEE Transactions on Image Processing"},{"key":"67_CR111","doi-asserted-by":"publisher","first-page":"1183","DOI":"10.1145\/3123266.3123405","volume-title":"Proceedings of the 2017 ACM on multimedia conference","author":"H. Mao","year":"2017","unstructured":"Mao, H., Cheung, M., & She, J. (2017). Deepart: learning joint representations of visual arts. In Q. Liu, R. Lienhart, H. Wang, et al. (Eds.), Proceedings of the 2017 ACM on multimedia conference (pp. 1183\u20131191). New York: ACM."},{"key":"67_CR112","doi-asserted-by":"publisher","first-page":"3108","DOI":"10.1145\/3477495.3531753","volume-title":"Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval","author":"P. Lerner","year":"2022","unstructured":"Lerner, P., Ferret, O., Guinaudeau, C., Le Borgne, H., Besan\u00e7on, R., Moreno, J. G., et al. (2022). ViQuAE, a dataset for knowledge-based visual question answering about named entities. In E. Amig\u00f3, P. Castells, J. Gonzalo, et al. (Eds.), Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval (pp. 3108\u20133120). New York: ACM."},{"issue":"12","key":"67_CR113","doi-asserted-by":"publisher","first-page":"8555","DOI":"10.1109\/TGRS.2020.2988782","volume":"58","author":"S. Lobry","year":"2020","unstructured":"Lobry, S., Marcos, D., Murray, J., & Tuia, D. (2020). RSVQA: visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 58(12), 8555\u20138566.","journal-title":"IEEE Transactions on Geoscience and Remote Sensing"},{"key":"67_CR114","doi-asserted-by":"crossref","unstructured":"Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., et\u00a0al. (2023). DriveLM: driving with graph visual question answering. arXiv preprint. arXiv:2312.14150.","DOI":"10.1007\/978-3-031-72943-0_15"},{"key":"67_CR115","unstructured":"Liu, Y., Li, Z., Li, H., Yu, W., Huang, M., Peng, D., et\u00a0al. (2023). On the hidden mystery of OCR in large multimodal models. arXiv preprint. arXiv:2305.07895."},{"key":"67_CR116","first-page":"9556","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"X. Yue","year":"2024","unstructured":"Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., et al. (2024). MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 9556\u20139567). Piscataway: IEEE."},{"key":"67_CR117","doi-asserted-by":"publisher","first-page":"11198","DOI":"10.1145\/3664647.3685520","volume-title":"Proceedings of the 32nd ACM international conference on multimedia","author":"H. Duan","year":"2024","unstructured":"Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., et al. (2024). VLMEvalKit: an open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia (pp. 11198\u201311201). New York: ACM."},{"key":"67_CR118","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., et\u00a0al. (2023). GPT-4 technical report. arXiv preprint. arXiv:2303.08774."},{"key":"67_CR119","unstructured":"Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J., et\u00a0al. (2024). Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv preprint. arXiv:2403.05530."},{"key":"67_CR120","unstructured":"Anthropic (2024). Claude 3.5 Sonnet model card addendum. Retrieved November 20, 2024, from https:\/\/www.anthropic.com."},{"key":"67_CR121","unstructured":"Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S.C., et\u00a0al. (2024). Cambrian-1: a fully open, vision-centric exploration of multimodal LLMs. arXiv preprint. arXiv:2406.16860."},{"key":"67_CR122","unstructured":"Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., et\u00a0al. (2024). Qwen2-VL: enhancing vision-language model\u2019s perception of the world at any resolution. arXiv preprint. arXiv:2409.12191."},{"key":"67_CR123","unstructured":"Zhang, Y.-F., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., et\u00a0al. (2024). MME-RealWorld: could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint. arXiv:2408.13257."},{"key":"67_CR124","doi-asserted-by":"crossref","unstructured":"Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., et\u00a0al. (2023). ShareGPT4V: improving large multi-modal models with better captions. arXiv preprint. arXiv:2311.12793.","DOI":"10.1007\/978-3-031-72643-9_22"},{"key":"67_CR125","first-page":"5648","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"K. Kafle","year":"2018","unstructured":"Kafle, K., Price, B., Cohen, S., & Kanan, C. (2018). DVQA: understanding data visualizations via question answering. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 5648\u20135656). Piscataway: IEEE."},{"key":"67_CR126","doi-asserted-by":"crossref","unstructured":"Li, J., & Lu, T. (2024). Driving with InternVL. Retrieved November 20, 2024, from https:\/\/opendrivelab.github.io\/Challenge.","DOI":"10.5465\/AMPROC.2024.303bp"},{"key":"67_CR127","first-page":"4015","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"A. Kirillov","year":"2023","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., et al. (2023). Segment anything. In Proceedings of the IEEE\/CVF international conference on computer vision (pp. 4015\u20134026). Piscataway: IEEE."},{"key":"67_CR128","unstructured":"Anthropic (2024). The Claude 3 model family: Opus, Sonnet, Haiku. Retrieved November 20, 2024, from https:\/\/www.anthropic.com."},{"key":"67_CR129","first-page":"577","volume-title":"Proceedings of the 15th European conference on computer vision","author":"J. Kim","year":"2018","unstructured":"Kim, J., Rohrbach, A., Darrell, T., Canny, J., & Akata, Z. (2018). Textual explanations for self-driving vehicles. In V. Ferrari, M. Hebert, C. Sminchisescu, et al. (Eds.), Proceedings of the 15th European conference on computer vision (pp. 577\u2013593). Cham: Springer."},{"key":"67_CR130","first-page":"4566","volume-title":"Proceedings of the IEEE conference on computer vision and pattern recognition","author":"R. Vedantam","year":"2015","unstructured":"Vedantam, R., Zitnick, C. L., & Parikh, D. (2015). CIDEr: consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4566\u20134575). Piscataway: IEEE."},{"key":"67_CR131","first-page":"311","volume-title":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics","author":"K. Papineni","year":"2002","unstructured":"Papineni, K., Roukos, S., Ward, T., & Zhu, W. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311\u2013318). Stroudsburg: ACL."},{"key":"67_CR132","first-page":"74","volume-title":"Proceedings of the ACL 2004 workshop on text summarization branches out","author":"C.-Y. Lin","year":"2004","unstructured":"Lin, C.-Y. (2004). ROUGE: a package for automatic evaluation of summaries. In Proceedings of the ACL 2004 workshop on text summarization branches out (pp. 74\u201381). Stroudsburg: ACL."},{"key":"67_CR133","first-page":"7554","volume-title":"Proceedings of IEEE international conference on robotics and automation","author":"B. Jin","year":"2023","unstructured":"Jin, B., Liu, X., Zheng, Y., Li, P., Zhao, H., Zhang, T., et al. (2023). ADAPT: action-aware driving caption transformer. In Proceedings of IEEE international conference on robotics and automation (pp. 7554\u20137561). Piscataway: IEEE."},{"key":"67_CR134","first-page":"525","volume-title":"Proceedings of the 26th international conference on medical image computing and computer-assisted intervention","author":"W. Lin","year":"2023","unstructured":"Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., et al. (2023). PMC-CLIP: contrastive language-image pre-training using biomedical documents. In H. Greenspan, A. Madabhushi, P. Mousavi, et al. (Eds.), Proceedings of the 26th international conference on medical image computing and computer-assisted intervention (pp. 525\u2013536). Stroudsburg: ACL."},{"key":"67_CR135","first-page":"2112","volume-title":"Findings of the Association for Computational Linguistics: empirical methods in natural language processing","author":"S. Subramanian","year":"2020","unstructured":"Subramanian, S., Wang, L. L., Mehta, S., Bogin, B., van Zuylen, M., Parasa, S., et al. (2020). MedICaT: a dataset of medical images, captions, and textual references. In T. Cohn, Y. He, & Y. Liu (Eds.), Findings of the Association for Computational Linguistics: empirical methods in natural language processing (pp. 2112\u20132120). Stroudsburg: ACL."},{"key":"67_CR136","unstructured":"Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023). Towards generalist foundation model for radiology. arXiv preprint. arXiv:2308.02463."},{"key":"67_CR137","first-page":"1","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"W.O. Ikezogwo","year":"2023","unstructured":"Ikezogwo, W.O., Seyfioglu, M. S., Ghezloo, F., Geva, D.S.C., Mohammed, F.S., Anand, P.K., et al. (2023). Quilt-1m: one million image-text pairs for histopathology. In A. Oh, T. Neumann, A. Globerson, et al. (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp. 1\u201323). Red Hook: Curran Associates."},{"issue":"1","key":"67_CR138","first-page":"32","volume":"25","author":"A. E. W. Johnson","year":"2018","unstructured":"Johnson, A. E. W., Stone, D. J., Celi, L. A., & Pollard, T. J. (2018). The MIMIC code repository: enabling reproducibility in critical care research. JAMIA, 25(1), 32\u201339.","journal-title":"JAMIA"},{"key":"67_CR139","unstructured":"Retina Image Bank (2024). Retrieved November 28, 2024, from https:\/\/imagebank.asrs.org\/."},{"key":"67_CR140","unstructured":"Chen, P., Ye, J., Wang, G., Li, Y., Deng, Z., Li, W., et\u00a0al. (2024). GMAI-MMBench: a comprehensive multimodal evaluation benchmark towards general medical AI. arXiv preprint. arXiv:2408.03361."},{"key":"67_CR141","unstructured":"Luo, J., Pang, Z., Zhang, Y., Wang, T., Wang, L., Dang, B., et\u00a0al. (2024). SkySenseGPT: a fine-grained instruction tuning dataset and model for remote sensing vision-language understanding. arXiv preprint. arXiv:2406.10100."},{"key":"67_CR142","first-page":"1","volume":"61","author":"Y. Zhan","year":"2023","unstructured":"Zhan, Y., Xiong, Z., & Yuan, Y. (2023). RSVG: exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 61, 1\u201313.","journal-title":"IEEE Transactions on Geoscience and Remote Sensing"},{"key":"67_CR143","unstructured":"Zhan, Y., Xiong, Z., & Yuan, Y. (2024). SkyEyeGPT: unifying remote sensing vision-language tasks via instruction tuning with large language model. arXiv preprint. arXiv:2401.09712."},{"key":"67_CR144","first-page":"1","volume":"61","author":"Z. Zhang","year":"2023","unstructured":"Zhang, Z., Jiao, L., Li, L., Liu, X., Chen, P., Liu, F., et al. (2023). A spatial hierarchical reasoning network for remote sensing visual question answering. IEEE Transactions on Geoscience and Remote Sensing, 61, 1\u201315.","journal-title":"IEEE Transactions on Geoscience and Remote Sensing"},{"key":"67_CR145","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TGRS.2022.3192460","volume":"60","author":"Y. Bazi","year":"2022","unstructured":"Bazi, Y., Al\u00a0Rahhal, M.M., Mekhalfi, M.L., Al\u00a0Zuair, M.A., & Melgani, F. (2022). Bi-modal transformer-based approach for visual question answering in remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 60, 1\u201311.","journal-title":"IEEE Transactions on Geoscience and Remote Sensing"},{"key":"67_CR146","first-page":"1","volume":"60","author":"Z. Yuan","year":"2022","unstructured":"Yuan, Z., Mou, L., Wang, Q., & Zhu, X.X. (2022). From easy to hard: learning language-guided curriculum for visual question answering on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 60, 1\u201311.","journal-title":"IEEE Transactions on Geoscience and Remote Sensing"},{"key":"67_CR147","unstructured":"Hu, E. J., Shen, Y., Wallis, P., Zhu, Z.A., Li, Y., Wang, S., et\u00a0al. (2021). LoRA: low-rank adaptation of large language models. arXiv preprint. arXiv:2106.09685."},{"key":"67_CR148","first-page":"1","volume-title":"Proceedings of the international conference for high performance computing, networking, storage and analysis","author":"S. Rajbhandari","year":"2020","unstructured":"Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). ZeRO: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, & W. T. Kramer (Eds.), Proceedings of the international conference for high performance computing, networking, storage and analysis (pp. 1\u201316). Piscataway: IEEE."},{"key":"67_CR149","first-page":"26296","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"H. Liu","year":"2024","unstructured":"Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved baselines with visual instruction tuning. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 26296\u201326306). Piscataway: IEEE."}],"container-title":["Visual Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44267-024-00067-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44267-024-00067-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44267-024-00067-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,10]],"date-time":"2024-12-10T11:23:31Z","timestamp":1733829811000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44267-024-00067-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,10]]},"references-count":149,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["67"],"URL":"https:\/\/doi.org\/10.1007\/s44267-024-00067-6","relation":{},"ISSN":["2731-9008"],"issn-type":[{"value":"2731-9008","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,10]]},"assertion":[{"value":"14 October 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 November 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 December 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 December 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Wenhai Wang is an Associate Editor at Visual Intelligence and was not involved in the editorial review of this article or the decision to publish it. The authors declare that they have no other competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"32"}}