{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T16:13:14Z","timestamp":1777047194817,"version":"3.51.4"},"reference-count":37,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T00:00:00Z","timestamp":1776988800000},"content-version":"vor","delay-in-days":113,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,10]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>This paper introduces MERLIN, a novel testbed system for the task of Multilingual Multimodal Entity Linking. The created dataset includes BBC news article titles, paired with corresponding images, in five languages: Hindi, Japanese, Indonesian, Vietnamese, and Tamil, featuring over 7,000 named entity mentions linked to 2,500 unique Wikidata entities. We also include several benchmarks using multilingual and multimodal entity linking methods exploring different language models like LLaMa-2 and Aya-23. Our findings indicate that incorporating visual data improves the accuracy of entity linking, especially for entities where the textual context is ambiguous or insufficient, and particularly for models that do not have strong multilingual abilities. For the work, the dataset, methods are available online.1<\/jats:p>","DOI":"10.1162\/tacl.a.633","type":"journal-article","created":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T15:19:36Z","timestamp":1777043976000},"page":"399-417","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking"],"prefix":"10.1162","volume":"14","author":[{"given":"Sathyanarayanan","family":"Ramamoorthy","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, USA. sramamoo@cs.cmu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vishwa","family":"Shah","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA. vishwavs@cs.cmu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Simran","family":"Khanuja","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA. skhanuja@cs.cmu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zaid","family":"Sheikh","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA. zsheikh@cs.cmu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shan","family":"Jie","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Agency, Singapore. yshanjie@dsta.gov.sg"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ann","family":"Chia","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Agency, Singapore. achiajin@dsta.gov.sg"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shearman","family":"Chua","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Agency, Singapore. schuawei@dsta.gov.sg"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Graham","family":"Neubig","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA. gneubig@cs.cmu.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2026,4,10]]},"reference":[{"key":"2026042411193154300_bib1","doi-asserted-by":"publisher","first-page":"4285","DOI":"10.1007\/978-3-030-45439-5_31","article-title":"Building a multimodal entity linking dataset from tweets","volume-title":"Proceedings of the Twelfth Language Resources and Evaluation Conference","author":"Adjali","year":"2020"},{"key":"2026042411193154300_bib2","article-title":"Aya 23: Open weight releases to further multilingual progress","author":"Aryabumi","year":"2024","journal-title":"arXiv preprint arXiv:2405.15032v2"},{"key":"2026042411193154300_bib3","doi-asserted-by":"publisher","first-page":"2478","DOI":"10.18653\/v1\/2022.acl-long.177","article-title":"ExtEnD: Extractive entity disambiguation","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Barba","year":"2022"},{"key":"2026042411193154300_bib4","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1145\/2684822.2685317","article-title":"Fast and space-efficient entity linking for queries","volume-title":"Proceedings of the Eighth ACM International Conference on Web Search and Data Mining","author":"Blanco","year":"2015"},{"key":"2026042411193154300_bib5","doi-asserted-by":"publisher","first-page":"5486","DOI":"10.18653\/v1\/2022.acl-long.376","article-title":"Systematic inequalities in language technology performance across the world\u2019s languages","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Blasi","year":"2022"},{"key":"2026042411193154300_bib6","doi-asserted-by":"publisher","first-page":"7833","DOI":"10.18653\/v1\/2020.emnlp-main.630","article-title":"Entity Linking in 100 Languages","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Botha","year":"2020"},{"key":"2026042411193154300_bib7","article-title":"No language left behind: Scaling human-centered machine translation","author":"Costa-juss\u00e0","year":"2022","journal-title":"arXiv preprint arXiv:2207.04672"},{"key":"2026042411193154300_bib8","doi-asserted-by":"publisher","first-page":"2306","DOI":"10.18653\/v1\/N19-1240","article-title":"Question answering by reasoning across documents with graph convolutional networks","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"De Cao","year":"2019"},{"key":"2026042411193154300_bib9","article-title":"Autoregressive entity retrieval","author":"De Cao","year":"2021","journal-title":"arXiv preprint arXiv:2010.00904v3"},{"key":"2026042411193154300_bib10","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1162\/tacl_a_00460","article-title":"Multilingual autoregressive entity linking","volume":"10","author":"De Cao","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026042411193154300_bib11","doi-asserted-by":"publisher","first-page":"4937","DOI":"10.18653\/v1\/2020.emnlp-main.400","article-title":"Entities as experts: Sparse memory access with entity supervision","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"F\u00e9vry","year":"2020"},{"key":"2026042411193154300_bib12","article-title":"Design challenges in low-resource cross-lingual entity linking","author":"Xingyu","year":"2020","journal-title":"arXiv preprint arXiv:2005.00692"},{"key":"2026042411193154300_bib13","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1145\/3474085.3475400","article-title":"Multimodal entity linking: A new dataset and a baseline","volume-title":"Proceedings of the 29th ACM International Conference on Multimedia","author":"Gan","year":"2021"},{"key":"2026042411193154300_bib14","first-page":"1020","article-title":"To link or not to link? A study on end-to-end tweet entity linking","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Guo","year":"2013"},{"key":"2026042411193154300_bib15","article-title":"Large multilingual models pivot zero-shot multimodal learning across languages","author":"Jinyi","year":"2023","journal-title":"arXiv preprint arXiv:2308.12038v3"},{"key":"2026042411193154300_bib16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.560","article-title":"The state and fate of linguistic diversity and inclusion in the NLP world","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Joshi","year":"2020"},{"key":"2026042411193154300_bib17","first-page":"5","article-title":"The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations","author":"Klie","year":"2018"},{"key":"2026042411193154300_bib18","doi-asserted-by":"publisher","first-page":"3696","DOI":"10.18653\/v1\/2022.findings-acl.292","article-title":"Improving candidate retrieval with entity profile generation for Wikidata entity linking","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Lai","year":"2022"},{"key":"2026042411193154300_bib19","doi-asserted-by":"publisher","first-page":"3449","DOI":"10.18653\/v1\/P19-1335","article-title":"Zero-shot entity linking by reading entity descriptions","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Logeswaran","year":"2019"},{"issue":"3","key":"2026042411193154300_bib20","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: The kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochemia Medica"},{"key":"2026042411193154300_bib21","first-page":"255","article-title":"Cross-language entity linking","volume-title":"Proceedings of 5th International Joint Conference on Natural Language Processing","author":"McNamee","year":"2011"},{"key":"2026042411193154300_bib22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1186","article-title":"Multimodal named entity disambiguation for noisy social media posts","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Moon","year":"2018"},{"key":"2026042411193154300_bib23","article-title":"Multilingual end to end entity linking","author":"Plekhanov","year":"2023","journal-title":"arXiv preprint arXiv:2306.08896v1"},{"key":"2026042411193154300_bib24","doi-asserted-by":"crossref","first-page":"7654","DOI":"10.63317\/48465grj4cjm","article-title":"Generative multimodal entity linking","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)","author":"Shi","year":"2024"},{"key":"2026042411193154300_bib25","doi-asserted-by":"publisher","first-page":"2255","DOI":"10.18653\/v1\/P16-1213","article-title":"One for all: Towards language independent named entity linking","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sil","year":"2016"},{"key":"2026042411193154300_bib26","article-title":"DWE+: Dual-way matching enhanced framework for multimodal entity linking","author":"Song","year":"2024","journal-title":"arXiv preprint arXiv:2404.04818v1"},{"key":"2026042411193154300_bib27","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"ArXiv"},{"key":"2026042411193154300_bib28","doi-asserted-by":"publisher","first-page":"3620","DOI":"10.18653\/v1\/2023.eacl-main.263","article-title":"Large scale multi-lingual multi-modal summarization dataset","volume-title":"Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics","author":"Verma","year":"2023"},{"issue":"8","key":"2026042411193154300_bib29","doi-asserted-by":"publisher","first-page":"6252","DOI":"10.1007\/s10489-024-05490-2","article-title":"2M-NER: Contrastive learning for multilingual and multimodal ner with language and modal fusion","volume":"54","author":"Wang","year":"2024","journal-title":"Applied Intelligence"},{"key":"2026042411193154300_bib30","doi-asserted-by":"publisher","first-page":"4785","DOI":"10.18653\/v1\/2022.acl-long.328","article-title":"WikiDiverse: A multimodal entity linking dataset with diversified contextual topics and entity types","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Wang","year":"2022"},{"key":"2026042411193154300_bib31","doi-asserted-by":"publisher","first-page":"6397","DOI":"10.18653\/v1\/2020.emnlp-main.519","article-title":"Scalable zero-shot entity linking with dense entity retrieval","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Ledell","year":"2020"},{"key":"2026042411193154300_bib32","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3612575","article-title":"DRIN: Dynamic relation interactive network for multimodal entity linking","author":"Xing","year":"2023","journal-title":"Proceedings of the 31st ACM International Conference on Multimedia"},{"key":"2026042411193154300_bib33","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1007\/978-3-031-17189-5_19","article-title":"Enhancing entity linking with contextualized entity embeddings","volume-title":"Natural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September 24\u201325, 2022, Proceedings, Part II","author":"Zhenran","year":"2022"},{"key":"2026042411193154300_bib34","doi-asserted-by":"publisher","first-page":"2816","DOI":"10.18653\/v1\/2024.eacl-long.172","article-title":"Ameli: Enhancing multimodal entity linking with fine-grained attributes","volume-title":"Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Yao","year":"2024"},{"key":"2026042411193154300_bib35","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1007\/978-3-030-73197-7_35","article-title":"Attention-based multimodal entity linking with high-quality images","volume-title":"Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11\u201314, 2021, Proceedings, Part II","author":"Li","year":"2021"},{"issue":"10","key":"2026042411193154300_bib36","doi-asserted-by":"publisher","first-page":"10098","DOI":"10.1109\/TKDE.2023.3250499","article-title":"Knowledge graph augmented network towards multiview representation learning for aspect-based sentiment analysis","volume":"35","author":"Zhong","year":"2023","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"2026042411193154300_bib37","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1162\/tacl_a_00303","article-title":"Improving candidate generation for low-resource cross-lingual entity linking","volume":"8","author":"Zhou","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.633\/2597128\/tacl.a.633.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.633\/2597128\/tacl.a.633.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T15:19:38Z","timestamp":1777043978000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.633\/136341\/MERLIN-A-Testbed-for-Multilingual-Multimodal"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":37,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.633","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}