{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,6]],"date-time":"2026-11-06T11:42:46Z","timestamp":1793965366927,"version":"4.8.0"},"reference-count":106,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T00:00:00Z","timestamp":1751328000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Commun"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Multimodal large language models have revolutionized AI research and industry, paving the way toward the next milestone. However, their large sizes and high computational costs restrict deployment to cloud servers, limiting use in mobile, offline, energy-sensitive, or privacy-critical scenarios. We present MiniCPM-V, efficient models for edge devices that integrate advancements in architecture, training, and data. The 8B model outperforms GPT-4V, Gemini Pro, and Claude 3 across 11 public benchmarks, processes high-resolution images at any aspect ratio, achieves robust optical character recognition, exhibits low hallucination rates, and supports over 30 languages while running efficiently on mobile phones. This progress reflects a broader trend: The sizes for high-performing models are rapidly decreasing alongside growing edge computation capacity, enabling advanced multimodal models to operate locally on consumer hardware. Such developments unlock applications across diverse real-world scenarios, from enhanced mobile AI to privacy-preserving solutions, marking a critical step toward democratizing powerful multimodal intelligence.<\/jats:p>","DOI":"10.1038\/s41467-025-61040-5","type":"journal-article","created":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T07:05:03Z","timestamp":1751353503000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":55,"title":["Efficient GPT-4V level multimodal large language model for deployment on edge devices"],"prefix":"10.1038","volume":"16","author":[{"given":"Yuan","family":"Yao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianyu","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ao","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chongyi","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junbo","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongji","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianchi","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chi","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoyu","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weilin","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhihui","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qianyu","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ronghua","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhensheng","family":"Zou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoye","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shengding","family":"Hu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhi","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xu","family":"Han","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guoyang","family":"Zeng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dahai","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7709-2543","authenticated-orcid":false,"given":"Zhiyuan","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6011-6115","authenticated-orcid":false,"given":"Maosong","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,7,1]]},"reference":[{"key":"61040_CR1","unstructured":"Reid, M. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint at https:\/\/arxiv.org\/abs\/2403.05530 (2024)."},{"key":"61040_CR2","unstructured":"Achiam, J. et al. GPT-4 Technical Report. Preprint at https:\/\/arxiv.org\/abs\/2303.08774 (2023)."},{"key":"61040_CR3","unstructured":"Lu, H. et al. DeepSeek-VL: Towards real-world vision-language understanding. Preprint at https:\/\/arxiv.org\/abs\/2403.05525 (2024)."},{"key":"61040_CR4","unstructured":"Liu, H., Li, C., Wu, Q. & Lee, Y. J. Visual instruction tuning. NeurIPS 36 (2024)."},{"key":"61040_CR5","unstructured":"Wang, W. et al. CogVLM: Visual expert for pretrained language models. In: Proc. NeurIPS (2023)."},{"key":"61040_CR6","doi-asserted-by":"crossref","unstructured":"Chen, Z. et al. How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites. In Science China Information Sciences (2024).","DOI":"10.1007\/s11432-024-4231-5"},{"key":"61040_CR7","unstructured":"Bai, J. et al. Qwen-VL: a frontier large vision-language model with versatile abilities. Preprint at https:\/\/arxiv.org\/abs\/2308.12966 (2023)."},{"key":"61040_CR8","unstructured":"Hu, J. et al. Large multilingual models pivot zero-shot multimodal learning across languages. In Proc. ICLR (2024)."},{"key":"61040_CR9","unstructured":"Lauren\u00e7on, H., Tronchon, L., Cord, M. & Sanh, V. What matters when building vision-language models? In Proc. NeurIPS (2024)."},{"key":"61040_CR10","doi-asserted-by":"crossref","unstructured":"McKinzie, B. et al. MM1: methods, analysis & insights from multimodal LLM pre-training. In Proc. ECCV (2024).","DOI":"10.1007\/978-3-031-73397-0_18"},{"key":"61040_CR11","unstructured":"Beyer, L. et al. PaliGemma: a versatile 3b vlm for transfer. Preprint at https:\/\/arxiv.org\/abs\/2407.07726 (2024)."},{"key":"61040_CR12","unstructured":"Abdin, M. et al. Phi-3 technical report: a highly capable language model locally on your phone. Preprint at https:\/\/arxiv.org\/abs\/2404.14219 (2024)."},{"key":"61040_CR13","unstructured":"Young, A. et al. Yi: open foundation models by 01.AI. Preprint at https:\/\/arxiv.org\/abs\/2403.04652 (2024)."},{"key":"61040_CR14","unstructured":"Li, Y. et al. Mini-Gemini: mining the potential of multi-modality vision language models. Preprint at https:\/\/arxiv.org\/abs\/2403.18814 (2024)."},{"key":"61040_CR15","unstructured":"Wikipedia. Moore\u2019s law. In Wikipedia, the Free Encyclopedia (2001)."},{"key":"61040_CR16","unstructured":"Wikipedia. Thrust\u2011to\u2011weight ratio. In Wikipedia, the Free Encyclopedia (2024)."},{"key":"61040_CR17","unstructured":"Xu, R. et al. LLaVA-UHD: an LMM perceiving any aspect ratio and high-resolution images. In Proc. ECCV (2024)."},{"key":"61040_CR18","unstructured":"Yu, T. et al. RLAIF-V: aligning MLLMs through open-source AI feedback for super GPT-4V trustworthiness. In Proc. CVPR (2025)."},{"key":"61040_CR19","doi-asserted-by":"crossref","unstructured":"Yu, T. et al. RLHF-V: towards trustworthy MLLMs via behavior alignment from fine-grained correctional human feedback. In: Proc. CVPR (2024).","DOI":"10.1109\/CVPR52733.2024.01310"},{"key":"61040_CR20","unstructured":"Bavishi, R. et al. Introducing our multimodal models. In Adept Blog (2023)."},{"key":"61040_CR21","unstructured":"Rafailov, R. et al. Direct preference optimization: your language model is secretly a reward model. NeurIPS. 36 (2024)."},{"key":"61040_CR22","unstructured":"OpenCompass Contributors. OpenCompass: a universal evaluation platform for foundation models. In GitHub repository (2023)."},{"key":"61040_CR23","unstructured":"Fu, C. et al. MME: a comprehensive evaluation benchmark for multimodal large language models. Preprint at https:\/\/arxiv.org\/abs\/2306.13394 (2023)."},{"key":"61040_CR24","unstructured":"Liu, Y. et al. MMBench: Is your multi-modal model an all-around player? Preprint at https:\/\/arxiv.org\/abs\/2307.06281 (2023)."},{"key":"61040_CR25","doi-asserted-by":"crossref","unstructured":"Yue, X. et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proc. CVPR (2024).","DOI":"10.1109\/CVPR52733.2024.00913"},{"key":"61040_CR26","unstructured":"Lu, P. et al. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In Proc. ICLR (2024)."},{"key":"61040_CR27","doi-asserted-by":"crossref","unstructured":"Liu, Y. et al. OCRBench: on the hidden mystery of OCR in large multimodal models. In Science China Information Sciences (2023).","DOI":"10.1007\/s11432-024-4235-6"},{"key":"61040_CR28","doi-asserted-by":"crossref","unstructured":"Singh, A. et al. Towards VQA models that can read. In: Proc. CVPR (2019).","DOI":"10.1109\/CVPR.2019.00851"},{"key":"61040_CR29","doi-asserted-by":"crossref","unstructured":"Mathew, M., Karatzas, D. & Jawahar, C. DocVQA: a dataset for VQA on document images. In: Proc. WACV (2021).","DOI":"10.1109\/WACV48630.2021.00225"},{"key":"61040_CR30","doi-asserted-by":"crossref","unstructured":"Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T. & Saenko, K. Object hallucination in image captioning. In Proc. EMNLP (2018).","DOI":"10.18653\/v1\/D18-1437"},{"key":"61040_CR31","unstructured":"Liu, Y. et al. TextMonkey: An OCR-free large multimodal model for understanding document. Preprint at https:\/\/arxiv.org\/abs\/2403.04473 (2024)."},{"key":"61040_CR32","unstructured":"He, M. et al. Efficient multimodal learning from data-centric perspective. Preprint at https:\/\/arxiv.org\/abs\/2402.11530 (2024)."},{"key":"61040_CR33","unstructured":"XTuner Contributors. XTuner: a toolkit for efficiently fine\u2011tuning LLM. In GitHub repository (2023)."},{"key":"61040_CR34","unstructured":"Li, B. et al. LLaVA\u2011NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild. In LLaVA blog (2024)."},{"key":"61040_CR35","unstructured":"Tong, S. et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In Proc. NeurIPS (2024)."},{"key":"61040_CR36","unstructured":"Liu, H. et al. LLaVA\u2011NeXT: Improved reasoning, OCR, and world knowledge. In LLaVA blog (January 30, 2024)."},{"key":"61040_CR37","unstructured":"Chu, X. et al. MobileVLM: a fast, reproducible and strong vision language assistant for mobile devices. Preprint at https:\/\/arxiv.org\/abs\/2312.16886 (2023)."},{"key":"61040_CR38","unstructured":"Anthropic. Introducing the next generation of Claude. In Claude 3 blog (2024)."},{"key":"61040_CR39","doi-asserted-by":"crossref","unstructured":"Liu, H., Li, C., Li, Y. & Lee, Y. J. Improved baselines with visual instruction tuning. In: Proc. CVPR (2024).","DOI":"10.1109\/CVPR52733.2024.02484"},{"key":"61040_CR40","unstructured":"Georgi\u202fGerganov and the llama.cpp Group. llama.cpp: LLM inference in C\/C++ with cross\u2011platform support and hardware-optimized quantization. In GitHub repository (2023)."},{"key":"61040_CR41","unstructured":"OpenAI. Hello\u202fGPT\u20114o: our new flagship, multimodal model for real-time reasoning across text, vision, and audio. In OpenAI blog (May\u202f13,\u202f2024)."},{"key":"61040_CR42","unstructured":"DeepMind. Project Astra: a research prototype exploring capabilities toward a universal AI assistant. In Google\u202fDeepMind Models (2024)."},{"key":"61040_CR43","first-page":"25278","volume":"35","author":"C Schuhmann","year":"2022","unstructured":"Schuhmann, C. et al. LAION-5B: An open large-scale dataset for training next generation image-text models. NeurIPS 35, 25278\u201325294 (2022).","journal-title":"NeurIPS"},{"key":"61040_CR44","unstructured":"Byeon, M. et al. COYO-700M: Image-text pair dataset. In GitHub repository (2022)."},{"key":"61040_CR45","unstructured":"Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https:\/\/arxiv.org\/abs\/2303.12712 (2023)."},{"key":"61040_CR46","unstructured":"Hu, S. et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In Proc. COLM (2024)."},{"key":"61040_CR47","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y. et al. Microsoft COCO: Common objects in context. In: ECCV (Eds. Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T.) 740\u2013755 (Springer, 2014).","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"61040_CR48","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","volume":"123","author":"R Krishna","year":"2017","unstructured":"Krishna, R. et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations\u201d. IJCV 123, 32\u201373 (2017).","journal-title":"IJCV"},{"key":"61040_CR49","doi-asserted-by":"crossref","unstructured":"Sharma, P., Ding, N., Goodman, S. & Soricut, R. Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL (2018).","DOI":"10.18653\/v1\/P18-1238"},{"key":"61040_CR50","doi-asserted-by":"crossref","unstructured":"Changpinyo, S., Sharma, P., Ding, N. & Soricut, R. Conceptual 12M: pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: CVPR (2021).","DOI":"10.1109\/CVPR46437.2021.00356"},{"key":"61040_CR51","unstructured":"Wu, J. et al. AI Challenger: a large-scale dataset for going deeper in image understanding. Preprint at https:\/\/arxiv.org\/abs\/1711.06475 (2017)."},{"key":"61040_CR52","first-page":"26418","volume":"35","author":"J Gu","year":"2022","unstructured":"Gu, J. et al. Wukong: a 100 million large-scale Chinese cross-modal pre-training benchmark. NeurIPS 35, 26418\u201326431 (2022).","journal-title":"NeurIPS"},{"key":"61040_CR53","unstructured":"Xie, C., Li, J. & Zhang, B. CCMB: A Large-scale Chinese Cross-modal Benchmark. Preprint at https:\/\/arxiv.org\/abs\/2205.03860 (2022)."},{"key":"61040_CR54","doi-asserted-by":"crossref","unstructured":"Srinivasan, K., Raman, K., Chen, J., Bendersky, M. & Najork, M. WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning. In: SIGIR 2443\u20132449 (2021).","DOI":"10.1145\/3404835.3463257"},{"key":"61040_CR55","doi-asserted-by":"crossref","unstructured":"Biten, A. F., Tito, R., Gomez, L., Valveny, E.& Karatzas, D. OCR-IDL: OCR annotations for industry document library dataset. In: ECCV (Eds. Avidan, S., Brostow, G., Ciss\u00e9, M., Farinella, G. M. & Hassner, T.). 241\u2013252 (Springer, 2022)..","DOI":"10.1007\/978-3-031-25069-9_16"},{"key":"61040_CR56","doi-asserted-by":"crossref","unstructured":"Gupta, A., Vedaldi, A. & Zisserman, A. Synthetic data for text localisation in natural images. In: CVPR 2315\u20132324 (2016).","DOI":"10.1109\/CVPR.2016.254"},{"key":"61040_CR57","unstructured":"Kim, G. et al. OCR-free document understanding transformer. In: ECCV (Eds. Avidan, S., Brostow, G., Ciss\u00e9, M., Farinella, G. M. & Hassner, T.) (2022)."},{"key":"61040_CR58","doi-asserted-by":"crossref","unstructured":"Li, L. et al. Multimodal ArXiv: a dataset for improving scientific comprehension of large vision-language models. In Proc. ACL (2024).","DOI":"10.18653\/v1\/2024.acl-long.775"},{"key":"61040_CR59","doi-asserted-by":"crossref","unstructured":"Plummer, B. A. et al. Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV 2641\u20132649 2015.","DOI":"10.1109\/ICCV.2015.303"},{"key":"61040_CR60","unstructured":"Gao, H. et al. Are you talking to a machine? dataset and methods for multilingual image question. NeurIPS 28 (2015)."},{"key":"61040_CR61","unstructured":"Lu, P. et al. IconQA: a new benchmark for abstract diagram understanding and visual language reasoning. In Proc. NeurIPS Datasets and Benchmarks Track (2021)."},{"key":"61040_CR62","doi-asserted-by":"crossref","unstructured":"Hudson, D. A. & Manning, C. D. GQA: a new dataset for real-world visual reasoning and compositional question answering. In: CVPR 6700\u20136709 (2019).","DOI":"10.1109\/CVPR.2019.00686"},{"key":"61040_CR63","doi-asserted-by":"crossref","unstructured":"Antol, S. et al. VQA: Visual question answering. In: ICCV 2425\u20132433 (2015).","DOI":"10.1109\/ICCV.2015.279"},{"key":"61040_CR64","doi-asserted-by":"crossref","unstructured":"Johnson, J. et al. CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. In: CVPR 2901\u20132910 (2017).","DOI":"10.1109\/CVPR.2017.215"},{"key":"61040_CR65","doi-asserted-by":"crossref","unstructured":"Gurari, D. et al. VizWiz Grand Challenge: answering visual questions from blind people. In: CVPR 3608\u20133617 (2018).","DOI":"10.1109\/CVPR.2018.00380"},{"key":"61040_CR66","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Groth, O., Bernstein, M. & Fei-Fei, L. Visual7W: Grounded question answering in images. In: CVPR (2016).","DOI":"10.1109\/CVPR.2016.540"},{"key":"61040_CR67","unstructured":"Ren, M., Kiros, R. & Zemel, R. Exploring models and data for image question answering. NeurIPS 28 (2015)."},{"key":"61040_CR68","doi-asserted-by":"crossref","unstructured":"Marino, K., Rastegari, M., Farhadi, A. & Mottaghi, R. OK-VQA: A visual question answering benchmark requiring external knowledge. In: CVPR 3195\u20133204 (2019).","DOI":"10.1109\/CVPR.2019.00331"},{"key":"61040_CR69","doi-asserted-by":"crossref","unstructured":"Schwenk, D., Khandelwal, A., Clark, C., Marino, K. & Mottaghi, R. A-OKVQA: A benchmark for visual question answering using world knowledge. In: ECCV (Eds. Avidan, S., Brostow, G., Ciss\u00e9, M., Farinella, G. M. & Hassner, T.) 146\u2013162 (Springer, 2022)..","DOI":"10.1007\/978-3-031-20074-8_9"},{"key":"61040_CR70","doi-asserted-by":"publisher","first-page":"8876","DOI":"10.1609\/aaai.v33i01.33018876","volume":"33","author":"S Shah","year":"2019","unstructured":"Shah, S., Mishra, A., Yadati, N. & Talukdar, P. P. KVQA: knowledge-aware visual question answering. AAAI 33, 8876\u20138884 (2019).","journal-title":"AAAI"},{"key":"61040_CR71","first-page":"2507","volume":"35","author":"P Lu","year":"2022","unstructured":"Lu, P. et al. Learn to explain: multimodal reasoning via thought chains for science question answering. NeurIPS 35, 2507\u20132521 (2022).","journal-title":"NeurIPS"},{"key":"61040_CR72","doi-asserted-by":"crossref","unstructured":"Yu, L., Poirson, P., Yang, S., Berg, A. C. & Berg, T. L. Modeling context in referring expressions. In: ECCV. (Eds. Leibe, B., Matas, J., Sebe, N. & Welling, M.). 69\u201385 (Springer, 2016).","DOI":"10.1007\/978-3-319-46475-6_5"},{"key":"61040_CR73","unstructured":"Du, Y. et al. What makes for good visual instructions? Synthesizing complex visual reasoning instructions for visual instruction tuning. In Proc. COLING (2025)."},{"key":"61040_CR74","doi-asserted-by":"crossref","unstructured":"Zellers, R., Bisk, Y., Farhadi, A. & Choi, Y. From recognition to cognition: visual commonsense reasoning. In: CVPR (2019).","DOI":"10.1109\/CVPR.2019.00688"},{"key":"61040_CR75","doi-asserted-by":"crossref","unstructured":"Suhr, A., Lewis, M., Yeh, J. & Artzi, Y. A corpus of natural language for visual reasoning. In: ACL 217\u2013223 (2017).","DOI":"10.18653\/v1\/P17-2034"},{"key":"61040_CR76","unstructured":"Liu, F. et al. Mitigating hallucination in large multi-modal models via robust instruction tuning. In Proc. ICLR (2024)."},{"key":"61040_CR77","doi-asserted-by":"crossref","unstructured":"Chen, J. et al. GeoQA: a geometric question answering benchmark towards multimodal numerical reasoning. In Proc. ACL Findings (2021).","DOI":"10.18653\/v1\/2021.findings-acl.46"},{"key":"61040_CR78","doi-asserted-by":"crossref","unstructured":"Cherian, A., Peng, K.-C., Lohit, S., Smith, K. A. & Tenenbaum, J. B. Are deep neural networks smarter than second graders? In: CVPR 10834\u201310844 (2023).","DOI":"10.1109\/CVPR52729.2023.01043"},{"key":"61040_CR79","doi-asserted-by":"crossref","unstructured":"Mishra, A., Shekhar, S., Singh, A. K. & Chakraborty, A. OCR-VQA: Visual question answering by reading text in images. In: ICDAR. (2019).","DOI":"10.1109\/ICDAR.2019.00156"},{"key":"61040_CR80","doi-asserted-by":"crossref","unstructured":"Biten, A. F. et al. Scene text visual question answering. In: CVPR 4291\u20134301 (2019).","DOI":"10.1109\/ICCV.2019.00439"},{"key":"61040_CR81","doi-asserted-by":"publisher","first-page":"13878","DOI":"10.1609\/aaai.v35i15.17635","volume":"35","author":"R Tanaka","year":"2021","unstructured":"Tanaka, R., Nishida, K. & Yoshida, S. VisualMRC: Machine reading comprehension on document image. AAAI 35, 13878\u201313888 (2021).","journal-title":"AAAI"},{"key":"61040_CR82","doi-asserted-by":"crossref","unstructured":"Kafle, K., Price, B., Cohen, S. & Kanan, C. DVQA: understanding data visualizations via question answering. In: CVPR 5648\u20135656 (2018).","DOI":"10.1109\/CVPR.2018.00592"},{"key":"61040_CR83","unstructured":"Kahou, S. E. et al. FigureQA: an annotated figure dataset for visual reasoning. In Proc. ICLR (2018)."},{"key":"61040_CR84","doi-asserted-by":"crossref","unstructured":"Masry, A., Long, D. X., Tan, J. Q., Joty, S. & Hoque, E. ChartQA: a benchmark for question answering about charts with visual and logical reasoning. In Proc. ACL Findings (2022).","DOI":"10.18653\/v1\/2022.findings-acl.177"},{"key":"61040_CR85","unstructured":"Svetlichnaya, S. DeepForm: understand structured documents at scale. In Weights & Biases report (2020)."},{"key":"61040_CR86","unstructured":"Chen, W. et al. TabFact: a large-scale dataset for table-based fact verification. In Proc. ICLR (2020)."},{"key":"61040_CR87","doi-asserted-by":"crossref","unstructured":"Mathew, M. et al. InfographicVQA. In: WACV 1697\u20131706 (2022).","DOI":"10.1109\/WACV51458.2022.00264"},{"key":"61040_CR88","doi-asserted-by":"crossref","unstructured":"Stanis\u0142awek, T. et al. Kleister: Key information extraction datasets involving long documents with complex layouts. In: ICDAR (Eds. Llad\u00f3s, J., Lopresti, D. & Uchida, S.). 564\u2013579 (Springer, 2021).","DOI":"10.1007\/978-3-030-86549-8_36"},{"key":"61040_CR89","doi-asserted-by":"crossref","unstructured":"Pasupat, P. & Liang, P. Compositional semantic parsing on semi-structured tables. In Proc. ACL (2015).","DOI":"10.3115\/v1\/P15-1142"},{"key":"61040_CR90","doi-asserted-by":"crossref","unstructured":"Ahmed, S., Jawade, B., Pandey, S., Setlur, S. & Govindaraju, V. RealCQA: Scientific chart question answering as a test-bed for first-order logic. In: ICDAR (Eds. Fink, G. A., Jain, R., Kise, K. & Zanibbi, R.). 66\u201383 (Springer, 2023).","DOI":"10.1007\/978-3-031-41682-8_5"},{"key":"61040_CR91","doi-asserted-by":"crossref","unstructured":"Kembhavi, A. et al. A diagram is worth a dozen images. In: ECCV (Eds. Leibe, B., Matas, J., Sebe, N. & Welling, M.). 235\u2013251 (Springer, 2016).","DOI":"10.1007\/978-3-319-46493-0_15"},{"key":"61040_CR92","unstructured":"Shin, A., Ushiku, Y. & Harada, T. The color of the cat is gray: 1 million full-sentences visual question answering (FSVQA). Preprint at https:\/\/arxiv.org\/abs\/1609.06657 (2016)."},{"key":"61040_CR93","doi-asserted-by":"crossref","unstructured":"Das, A. et al. Visual Dialog. In: CVPR 326\u2013335 (2017).","DOI":"10.1109\/CVPR.2017.121"},{"key":"61040_CR94","unstructured":"Zhang, Y. et al. LLaVAR: enhanced visual instruction tuning for text-rich image understanding. Preprint at https:\/\/arxiv.org\/abs\/2306.17107 (2023)."},{"key":"61040_CR95","unstructured":"Carter, J. TextOCR\u2011GPT4V. In Hugging Face dataset repository (2024)."},{"key":"61040_CR96","unstructured":"Zhao, B., Wu, B. & Huang, T. SVIT: scaling up visual instruction tuning. Preprint at https:\/\/arxiv.org\/abs\/2307.04087 (2023)."},{"key":"61040_CR97","unstructured":"Yu, T. et al. Reformulating vision-language foundation models and datasets towards universal multimodal assistants. Preprint at https:\/\/arxiv.org\/abs\/2310.00653 (2023)."},{"key":"61040_CR98","doi-asserted-by":"crossref","unstructured":"Chen, L. et al. ShareGPT4V: Improving large multi-modal models with better captions. In Proc. ECCV (2024).","DOI":"10.1007\/978-3-031-72643-9_22"},{"key":"61040_CR99","doi-asserted-by":"crossref","unstructured":"Gupta, A., Dollar, P.& Girshick, R. LVIS: a dataset for large vocabulary instance segmentation. In: CVPR 5356\u20135364 (2019).","DOI":"10.1109\/CVPR.2019.00550"},{"key":"61040_CR100","unstructured":"Chen, G. H. et al. ALLaVA: harnessing GPT4V-synthesized data for a lite vision-language model. Preprint at https:\/\/arxiv.org\/abs\/2402.11684 (2024)."},{"key":"61040_CR101","doi-asserted-by":"crossref","unstructured":"Ding, N. et al. Enhancing chat language models by scaling high-quality instructional conversations. In Proc. EMNLP (2023).","DOI":"10.18653\/v1\/2023.emnlp-main.183"},{"key":"61040_CR102","unstructured":"Taori, R. et al. Stanford\u202fAlpaca: An instruction\u2011following LLaMA model. In GitHub repository (2023)."},{"key":"61040_CR103","unstructured":"Zheng, L. et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 36 (2024)."},{"key":"61040_CR104","unstructured":"BELLEGroup. BELLE: Be Everyone\u2019s Large Language model Engine. In GitHub repository (2023)."},{"key":"61040_CR105","unstructured":"Lian, W. et al. OpenOrca: an open dataset of GPT\u2011augmented FLAN reasoning traces. In Hugging Face dataset repository (2023)."},{"key":"61040_CR106","unstructured":"Teknium. OpenHermes\u202f2.5: An open dataset of synthetic data for generalist LLM assistants. In Hugging Face dataset repository (2023)."}],"container-title":["Nature Communications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41467-025-61040-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-025-61040-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-025-61040-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T07:05:35Z","timestamp":1751353535000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41467-025-61040-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,1]]},"references-count":106,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["61040"],"URL":"https:\/\/doi.org\/10.1038\/s41467-025-61040-5","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-5830327\/v1","asserted-by":"object"}]},"ISSN":["2041-1723"],"issn-type":[{"value":"2041-1723","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,1]]},"assertion":[{"value":"14 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 June 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"5509"}}