{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T07:04:37Z","timestamp":1779779077647,"version":"3.53.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T00:00:00Z","timestamp":1779753600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T00:00:00Z","timestamp":1779753600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Universit\u00e0 degli Studi di Bari Aldo Moro"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Large Vision-Language Models are an extension of Large Language Models to process signals from the vision-language domain. Their performance is remarkable since they exploit the cutting-edge capabilities of pre-trained Large Language Models. However, most works using these models focus on English language data for training and evaluation. In light of this, several works have been released to extend current training mixtures to consider non-English data. As a matter of fact, several works also considered cultural aspects to decrease bias related to American and Chinese culture. However, evaluation in non-English languages is still limited w.r.t. English. Specifically, there are no datasets that focus on text-centric non-English images, since most available multimodal benchmarks focus on natural images. This limitation makes it unclear whether Large Vision-Language Models can be used for non-English text-centric tasks. In this work, we aim to understand the capabilities of these models for the Italian language and introduce a benchmark dataset focusing on text-centric Italian images. Therefore, we introduce our benchmark dataset\n                    <jats:sc>ETCII<\/jats:sc>\n                    (\n                    <jats:italic>Evaluation of Text-Centric Italian Images<\/jats:italic>\n                    ), to evaluate Large Vision-Language Models on three diverse task categories for text-centric images.\n                  <\/jats:p>","DOI":"10.1007\/s10579-026-09926-8","type":"journal-article","created":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T06:46:14Z","timestamp":1779777974000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Do you understand Italian? Evaluating LVLMs on Italian visual question-answering"],"prefix":"10.1007","volume":"60","author":[{"given":"Elio","family":"Musacchio","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucia","family":"Siciliani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pierpaolo","family":"Basile","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giovanni","family":"Semeraro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,26]]},"reference":[{"key":"9926_CR1","doi-asserted-by":"publisher","unstructured":"Aynetdinov, A., & Akbik, A. (2024). Semscore: Automated evaluation of instruction-tuned llms based on semantic textual similarity. CoRR\u00a0abs\/2401.17072. https:\/\/doi.org\/10.48550\/ARXIV.2401.17072.","DOI":"10.48550\/ARXIV.2401.17072"},{"key":"9926_CR2","doi-asserted-by":"publisher","unstructured":"Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., & Lin, J. (2025). Qwen2.5-vl technical report. CoRR\u00a0abs\/2502.13923. https:\/\/doi.org\/10.48550\/ARXIV.2502.13923.","DOI":"10.48550\/ARXIV.2502.13923"},{"key":"9926_CR3","doi-asserted-by":"crossref","unstructured":"Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., & Karatzas, D. (2019). Scene text visual question answering. In Proceedings of the IEEE\/CVF international conference on computer vision, pp. 4291\u20134301.","DOI":"10.1109\/ICCV.2019.00439"},{"key":"9926_CR4","doi-asserted-by":"crossref","unstructured":"Chen, D., Liu, J., Dai, W., & Wang, B. (2024). Visual instruction tuning with polite flamingo. In Wooldridge, M.J., Dy, J.G., & Natarajan, S. (Eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 17745\u201317753. AAAI Press.","DOI":"10.1609\/aaai.v38i16.29727"},{"key":"9926_CR5","doi-asserted-by":"crossref","unstructured":"Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., & Lin, D. (2024). Sharegpt4v: Improving large multi-modal models with better captions. Lecture Notes in Computer Science. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision - ECCV 2024\u201318th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XVII (Vol. 15075, pp. 370\u2013387). Springer.","DOI":"10.1007\/978-3-031-72643-9_22"},{"key":"9926_CR6","doi-asserted-by":"publisher","unstructured":"Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., Gu, L., Wang, X., Li, Q., Ren, Y., Chen, Z., Luo, J., Wang, J., Jiang, T., Wang, B., He, C., Shi, B., Zhang, X., Lv, H., Wang, Y., Shao, W., Chu, P., Tu, Z., He, T., Wu, Z., Deng, H., Ge, J., Chen, K., Dou, M., Lu, L., Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., & Wang, W. (2024). Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. CoRR\u00a0abs\/2412.05271. https:\/\/doi.org\/10.48550\/ARXIV.2412.05271.","DOI":"10.48550\/ARXIV.2412.05271"},{"key":"9926_CR7","doi-asserted-by":"crossref","unstructured":"Croce, D., Passaro, L. C., Lenci, A., & Basili, R. (2021). Gqa-it: Italian question answering on image scene graphs. In E.\u00a0Fersini, M.\u00a0Passarotti, and V.\u00a0Patti (Eds.), Proceedings of the Eighth Italian Conference on Computational Linguistics, CLiC-it 2021, Milan, Italy, January 26-28, 2022, Volume 3033 of CEUR Workshop Proceedings. CEUR-WS.org.","DOI":"10.4000\/books.aaccademia.10535"},{"key":"9926_CR8","doi-asserted-by":"crossref","unstructured":"Das, R. J., Hristov, S. E., Li, H., Dimitrov, D., Koychev, I., & Nakov, P. (2024). EXAMS-V: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. In L.\u00a0Ku, A.\u00a0Martins, and V.\u00a0Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 7768\u20137791. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.acl-long.420"},{"key":"9926_CR9","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net."},{"key":"9926_CR10","doi-asserted-by":"publisher","unstructured":"Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., Li, K., Sun, X., & Ji, R. (2023). MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR\u00a0abs\/2306.13394. https:\/\/doi.org\/10.48550\/ARXIV.2306.13394.","DOI":"10.48550\/ARXIV.2306.13394"},{"key":"9926_CR11","doi-asserted-by":"publisher","unstructured":"Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., ... & Ma, Z. (2024). The llama 3 herd of models. CoRR\u00a0abs\/2407.21783. https:\/\/doi.org\/10.48550\/ARXIV.2407.21783.","DOI":"10.48550\/ARXIV.2407.21783"},{"key":"9926_CR12","doi-asserted-by":"crossref","unstructured":"Hudson, D. A., & Manning, C. D. (2019). GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700\u20136709. Computer Vision Foundation \/ IEEE.","DOI":"10.1109\/CVPR.2019.00686"},{"key":"9926_CR13","doi-asserted-by":"publisher","unstructured":"Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram\u00e9, A., Rivi\u00e8re, M., Rouillard, L., Mesnard, T., Cideron, G., Grill, J., Ramos, S., Yvinec, E., Casbon, M., Pot, E., Penchev, I., Liu, G., ... & Nardini, I. (2025). Gemma 3 technical report. CoRR\u00a0abs\/2503.19786. https:\/\/doi.org\/10.48550\/ARXIV.2503.19786.","DOI":"10.48550\/ARXIV.2503.19786"},{"key":"9926_CR16","doi-asserted-by":"crossref","unstructured":"Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved baselines with visual instruction tuning. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 26286\u201326296. IEEE.","DOI":"10.1109\/CVPR52733.2024.02484"},{"key":"9926_CR17","unstructured":"Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). (2023). Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023 (pp. 10\u201316). LA, USA, December."},{"key":"9926_CR14","doi-asserted-by":"crossref","unstructured":"Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., & Wen, J. R. (2023). Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.","DOI":"10.18653\/v1\/2023.emnlp-main.20"},{"key":"9926_CR15","doi-asserted-by":"crossref","unstructured":"Li, Z., Mondal, I., Nghiem, H., Liang, Y., & Boyd-Graber, J. L. (2024). PEDANTS: Cheap but effective and interpretable answer equivalence. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA (pp. 9373\u20139398). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.findings-emnlp.548"},{"key":"9926_CR18","unstructured":"Magnini, B., Zanoli, R., Resta, M., Cimmino, M., Albano, P., Madeddu, M., & Patti, V. (2025). Evalita-LLM: Benchmarking large language models on Italian.\u00a0arXiv preprint\u00a0arXiv:2502.02289."},{"key":"9926_CR19","doi-asserted-by":"crossref","unstructured":"Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019). Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE\/cvf conference on computer vision and pattern recognition, pp. 3195\u20133204.","DOI":"10.1109\/CVPR.2019.00331"},{"key":"9926_CR20","doi-asserted-by":"crossref","unstructured":"Mathew, M., Karatzas, D., Jawahar, C. V. (2021). Docvqa: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pp. 2199\u20132208. IEEE.","DOI":"10.1109\/WACV48630.2021.00225"},{"key":"9926_CR21","unstructured":"Musacchio, E., Siciliani, L., Basile, P., Semeraro, G. (2024). Llava-ndino: Empowering llms with multimodality for the italian language. In G.\u00a0Bonetta, C.\u00a0D. Hromei, L.\u00a0Siciliani, and M.\u00a0A. Stranisci (Eds.), Proceedings of the Eight Workshop on Natural Language for Artificial Intelligence (NL4AI 2024) co-located with 23th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2024), Bolzano, Italy, November 26th-27th, 2024, Volume 3877 of CEUR Workshop Proceedings. CEUR-WS.org."},{"key":"9926_CR22","doi-asserted-by":"publisher","unstructured":"OpenAI. (2023). GPT-4 technical report. CoRR\u00a0abs\/2303.08774. https:\/\/doi.org\/10.48550\/ARXIV.2303.08774.","DOI":"10.48550\/ARXIV.2303.08774"},{"key":"9926_CR23","unstructured":"Scaiella, A., Margiotta, D., Hromei, C. D., Croce, D., & Basili, R. (2024). Evaluating multimodal large language models for visual question-answering in italian. In Proceedings of the Eighth Workshop on Natural Language for Artificial Intelligence (NL4AI 2024) co-located with 23th International Conference of the Italian Association for Artificial Intelligence (AI* IA 2024), CEUR-WS. org."},{"key":"9926_CR24","doi-asserted-by":"crossref","unstructured":"Seveso, A., Potert\u00ec, D., Federici, E., Mezzanzanica, M., & Mercorio, F. (2025). ITALIC: An Italian culture-aware natural language benchmark. In L.\u00a0Chiruzzo, A.\u00a0Ritter, and L.\u00a0Wang (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Albuquerque, New Mexico, pp. 1469\u20131478. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2025.naacl-long.68"},{"key":"9926_CR25","doi-asserted-by":"crossref","unstructured":"Singh, S., Vargus, F., D\u2019souza, D., Karlsson, B., Mahendiran, A., Ko, W., Shandilya, H., Patel, J., Mataciunas, D., O\u2019Mahony, L., Zhang, M., Hettiarachchi, R., Wilson, J., Machado, M., Moura, L. S., Krzeminski, D., Fadaei, H., Erg\u00fcn, I., Okoh, I.,\u00a0\u2026\u00a0& Hooker, S. (2024). Aya dataset: An open-access collection for multilingual instruction tuning. In L. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11521\u201311567. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.acl-long.620"},{"key":"9926_CR26","doi-asserted-by":"publisher","unstructured":"Tang, J., Liu, Q., Ye, Y., Lu, J., Wei, S., Lin, C., Li, W., Mahmood, M. F. F. B., Feng, H., Zhao, Z., Wang, Y., Liu, Y., Liu, H., Bai, X., & Huang, C. (2024). MTVQA: benchmarking multilingual text-centric visual question answering. CoRR\u00a0abs\/2405.11985. https:\/\/doi.org\/10.48550\/ARXIV.2405.11985.","DOI":"10.48550\/ARXIV.2405.11985"},{"key":"9926_CR27","doi-asserted-by":"crossref","unstructured":"Testa, D., Bonetta, G., Bernardi, R., Bondielli, A., Lenci, A., Miaschi, A., Passaro, L., & Magnini, B. (2025). All-in-one: Understanding and generation in multimodal reasoning with the MAIA benchmark. In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China (pp. 20030\u201320050). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2025.findings-emnlp.1091"},{"key":"9926_CR28","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In I.\u00a0Guyon, U.\u00a0von Luxburg, S.\u00a0Bengio, H.\u00a0M. Wallach, R.\u00a0Fergus, S.\u00a0V.\u00a0N. Vishwanathan, and R.\u00a0Garnett (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998\u20136008."},{"key":"9926_CR29","doi-asserted-by":"crossref","unstructured":"Xu, Z., Shen, Y., & Huang, L. (2023). Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning. In A. Rogers, J. L. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 11445\u201311465. Association for Computational Linguistics.","DOI":"10.18653\/v1\/2023.acl-long.641"},{"key":"9926_CR30","doi-asserted-by":"publisher","unstructured":"Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., Chen, Q., Zhou, H., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., & Sun, M. (2024). Minicpm-v: A GPT-4V level MLLM on your phone. CoRR\u00a0abs\/2408.01800. https:\/\/doi.org\/10.48550\/ARXIV.2408.01800.","DOI":"10.48550\/ARXIV.2408.01800"},{"key":"9926_CR31","doi-asserted-by":"publisher","unstructured":"Yue, X., Song, Y., Asai, A., Kim, S., de Dieu Nyandwi, J., Khanuja, S., Kantharuban, A., Sutawika, L., Ramamoorthy, S., & Neubig, G. (2024). Pangea: A fully open multilingual multimodal LLM for 39 languages. CoRR\u00a0abs\/2410.16153. https:\/\/doi.org\/10.48550\/ARXIV.2410.16153.","DOI":"10.48550\/ARXIV.2410.16153"},{"key":"9926_CR32","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net."},{"key":"9926_CR33","first-page":"46595","volume":"36","author":"L Zheng","year":"2023","unstructured":"Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36, 46595\u201346623.","journal-title":"Advances in neural information processing systems"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-026-09926-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-026-09926-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-026-09926-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T06:46:30Z","timestamp":1779777990000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-026-09926-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,26]]},"references-count":33,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9]]}},"alternative-id":["9926"],"URL":"https:\/\/doi.org\/10.1007\/s10579-026-09926-8","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,26]]},"assertion":[{"value":"21 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 April 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), Spoke 6 - Symbiotic AI (CUP H97G22000210007) under the NRRP MUR program funded by the NextGenerationEU.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interest"}}],"article-number":"49"}}