{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T17:41:32Z","timestamp":1777398092254,"version":"3.51.4"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"8","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:p>Multimodal Entity Linking (MEL), which involves disambiguating a mention composed of multimodal inputs to a multimodal knowledge base (KB), has gained increasing attention. Although existing MEL approaches using supervised learning show promising performance, they depend heavily on large-scale labeled training data, which is expensive to obtain for each new scenario. Unsupervised learning MEL methods, on the other hand, typically consist of two main steps. In the first multimodal data encoding step, these methods either assume that the multimodal data inputs are of high quality or attempt to filter out the noisy modality. In the second entity ranking step, they employ a bipartite graph to model the relationships only between mentions and entities. However, unsupervised methods face challenges in both steps. In the first step, data quality issues arise, including limited context in textual inputs and noise in the corresponding images. Moreover, in the second step, the bipartite graph fails to capture coherence between highly correlated entities within the KB, which offers clues on shared domains among entities. This limitation hinders effective retrieval of the target entity. To address these issues, we propose a novel unsupervised learning framework, OpenMEL, for solving the MEL task. We enhance the textual modality contextual information by incorporating full context comprehension and general knowledge, and generates three levels of visual inputs for further adaptive selection to handle noise. To capture global entity coherence, we construct a tree cover structure, defining it as a maximum spanning tree with bounded nodes to meet the MEL objective. We then introduce a greedy algorithm with theoretical guarantees to solve this problem. Experimental results on three public benchmark datasets show that OpenMEL outperforms various state-of-the-art baselines.<\/jats:p>","DOI":"10.14778\/3742728.3742740","type":"journal-article","created":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T13:32:53Z","timestamp":1756906373000},"page":"2454-2467","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["OpenMEL: Unsupervised Multimodal Entity Linking Using Noise-Free Expanded Queries and Global Coherence"],"prefix":"10.14778","volume":"18","author":[{"given":"Xinyi","family":"Zhu","sequence":"first","affiliation":[{"name":"The Hong Kong University of Science and Technology (Guangzhou), Data Science and Analytics"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongqi","family":"Zhang","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology (Guangzhou), Data Science and Analytics"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Chen","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology (Guangzhou), Data Science and Analytics"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,3]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the Twelfth Language Resources and Evaluation Conference. 4285\u20134292","author":"Adjali Omar","year":"2020","unstructured":"Omar Adjali, Romaric Besan\u00e7on, Olivier Ferret, Herv\u00e9 Le Borgne, and Brigitte Grau. 2020. Building a multimodal entity linking dataset from tweets. In Proceedings of the Twelfth Language Resources and Evaluation Conference. 4285\u20134292."},{"key":"e_1_2_1_2_1","volume-title":"Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26","author":"Bordes Antoine","year":"2013","unstructured":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013)."},{"key":"e_1_2_1_3_1","volume-title":"Language models are few-shot learners. arXiv preprint arXiv:2005.14165","author":"Brown Tom B","year":"2020","unstructured":"Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)."},{"key":"e_1_2_1_4_1","volume-title":"Neural collective entity linking. arXiv preprint arXiv:1811.08603","author":"Cao Yixin","year":"2018","unstructured":"Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Neural collective entity linking. arXiv preprint arXiv:1811.08603 (2018)."},{"key":"e_1_2_1_5_1","volume-title":"Bridge text and knowledge by learning multi-prototype entity mention embedding","author":"Cao Yixin","unstructured":"Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 2017. Bridge text and knowledge by learning multi-prototype entity mention embedding. Association for Computational Linguistics."},{"key":"e_1_2_1_6_1","volume-title":"Kevin Chen-Chuan Chang, et al","author":"Cheng Tao","year":"2007","unstructured":"Tao Cheng, Kevin Chen-Chuan Chang, et al. 2007. Entity Search Engine: Towards Agile Best-Effort Information Integration over the Web. Ph.D. Dissertation. University of Illinois at Urbana-Champaign."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL). 708\u2013716","author":"Cucerzan Silviu","year":"2007","unstructured":"Silviu Cucerzan. 2007. Large-scale named entity disambiguation based on Wikipedia data. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL). 708\u2013716."},{"key":"e_1_2_1_8_1","volume-title":"Opentapioca: Lightweight entity linking for wikidata. arXiv preprint arXiv:1904.09131","author":"Delpeuch Antonin","year":"2019","unstructured":"Antonin Delpeuch. 2019. Opentapioca: Lightweight entity linking for wikidata. arXiv preprint arXiv:1904.09131 (2019)."},{"key":"e_1_2_1_9_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1873781.1873813"},{"key":"e_1_2_1_11_1","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings, Part I 17","author":"Dubey Mohnish","year":"2018","unstructured":"Mohnish Dubey, Debayan Banerjee, Debanjan Chaudhuri, and Jens Lehmann. 2018. EARL: joint entity and relation linking for question answering over knowledge graphs. In The Semantic Web-ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, October 8\u201312, 2018, Proceedings, Part I 17. Springer, 108\u2013126."},{"key":"e_1_2_1_13_1","volume-title":"Named entity disambiguation for noisy text. arXiv preprint arXiv:1706.09147","author":"Eshel Yotam","year":"2017","unstructured":"Yotam Eshel, Noam Cohen, Kira Radinsky, Shaul Markovitch, Ikuya Yamada, and Omer Levy. 2017. Named entity disambiguation for noisy text. arXiv preprint arXiv:1706.09147 (2017)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE53745.2022.00052"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Zheng Fang Yanan Cao Qian Li Dongjie Zhang Zhenyu Zhang and Yanbing Liu. 2019. Joint entity linking with deep reinforcement learning. In The world wide web conference. 438\u2013447.","DOI":"10.1145\/3308558.3313517"},{"key":"e_1_2_1_16_1","volume-title":"Capturing semantic similarity for entity linking with convolutional neural networks. arXiv preprint arXiv:1604.00734","author":"Francis-Landau Matthew","year":"2016","unstructured":"Matthew Francis-Landau, Greg Durrett, and Dan Klein. 2016. Capturing semantic similarity for entity linking with convolutional neural networks. arXiv preprint arXiv:1604.00734 (2016)."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475400"},{"key":"e_1_2_1_18_1","volume-title":"Deep joint entity disambiguation with local neural attention. arXiv preprint arXiv:1704.04920","author":"Ganea Octavian-Eugen","year":"2017","unstructured":"Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. arXiv preprint arXiv:1704.04920 (2017)."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 45th international acm sigir conference on research and development in information retrieval. 1455\u20131465","author":"Gerritse Emma J","year":"2022","unstructured":"Emma J Gerritse, Faegheh Hasibi, and Arjen P de Vries. 2022. Entity-aware transformers for entity search. In Proceedings of the 45th international acm sigir conference on research and development in information retrieval. 1455\u20131465."},{"key":"e_1_2_1_20_1","unstructured":"Dorit S Hochbaum. 1996. Approximating covering and packing problems: set cover vertex cover independent set and related problems. In Approximation algorithms for NP-hard problems. 94\u2013143."},{"key":"e_1_2_1_21_1","volume-title":"Evaluating pretrained transformer models for entity linking in task-oriented dialog. arXiv preprint arXiv:2112.08327","author":"Jayanthi Sai Muralidhar","year":"2021","unstructured":"Sai Muralidhar Jayanthi, Varsha Embar, and Karthik Raghunathan. 2021. Evaluating pretrained transformer models for entity linking in task-oriented dialog. arXiv preprint arXiv:2112.08327 (2021)."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies. 1148\u20131158","author":"Ji Heng","year":"2011","unstructured":"Heng Ji and Ralph Grishman. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies. 1148\u20131158."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9939-1956-0078686-7"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2021 International Conference on Management of Data. 1142\u20131155","author":"Lin Xueling","year":"2021","unstructured":"Xueling Lin, Lei Chen, and Chaorui Zhang. 2021. Tenet: Joint entity and relation linking with coherence relaxation. In Proceedings of the 2021 International Conference on Management of Data. 1142\u20131155."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/3384345.3384352"},{"key":"e_1_2_1_26_1","volume-title":"Zero-shot entity linking by reading entity descriptions. arXiv preprint arXiv:1906.07348","author":"Logeswaran Lajanugen","year":"2019","unstructured":"Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. arXiv preprint arXiv:1906.07348 (2019)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599439"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312677"},{"key":"e_1_2_1_29_1","volume-title":"Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26","author":"Mikolov Tomas","year":"2013","unstructured":"Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Bhaskar Mitra Nick Craswell et al. 2018. An introduction to neural information retrieval. Foundations and Trends\u00ae in Information Retrieval 13 1 (2018) 1\u2013126.","DOI":"10.1561\/1500000061"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1186"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the VLDB Endowment 11","author":"Nguyen Dat Ba","year":"2017","unstructured":"Dat Ba Nguyen, Abdalghani Abujabal, Khanh Tran, Martin Theobald, and Gerhard Weikum. 2017. Query-driven on-the-fly knowledge base construction. Proceedings of the VLDB Endowment 11, 1 (2017)."},{"key":"e_1_2_1_33_1","volume-title":"VLDB'02: Proceedings of the 28th International Conference on Very Large Databases. Elsevier, 1008\u20131018","author":"Ohura Yusuke","year":"2002","unstructured":"Yusuke Ohura, Katsumi Takahashi, Iko Pramudiono, and Masaru Kitsuregawa. 2002. Experiments on query expansion for internet yellow page services using web log mining. In VLDB'02: Proceedings of the 28th International Conference on Very Large Databases. Elsevier, 1008\u20131018."},{"key":"e_1_2_1_34_1","volume-title":"Bipartite graph matching algorithms for clean-clean entity resolution: an empirical evaluation. arXiv preprint arXiv:2112.14030","author":"Papadakis George","year":"2021","unstructured":"George Papadakis, Vasilis Efthymiou, Emanouil Thanos, and Oktie Hassanzadeh. 2021. Bipartite graph matching algorithms for clean-clean entity resolution: an empirical evaluation. arXiv preprint arXiv:2112.14030 (2021)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3327964.3328499"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623732"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1957.tb01515.x"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Priya Radhakrishnan","year":"2018","unstructured":"Radhakrishnan Priya, Talukdar Partha, Varma Vasudeva, et al. 2018. ELDEN: Improved entity linking using densified knowledge graphs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA. 1\u20136."},{"key":"e_1_2_1_39_1","volume-title":"International conference on machine learning. PMLR, 8748\u20138763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748\u20138763."},{"key":"e_1_2_1_40_1","volume-title":"A survey of statistical approaches for query expansion. Knowledge and information systems 61","author":"Raza Muhammad Ahsan","year":"2019","unstructured":"Muhammad Ahsan Raza, Rahmah Mokhtar, and Noraziah Ahmad. 2019. A survey of statistical approaches for query expansion. Knowledge and information systems 61 (2019), 1\u201325."},{"key":"e_1_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Ridho Reinanda Edgar Meij Maarten de Rijke et al. 2020. Knowledge graphs: An information retrieval perspective. Foundations and Trends\u00ae in Information Retrieval 14 4 (2020) 289\u2013444.","DOI":"10.1561\/1500000063"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1243"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687642"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 57th annual meeting of the association for computational linguistics: student research workshop. 315\u2013322","author":"Sevgili \u00d6zge","year":"2019","unstructured":"\u00d6zge Sevgili, Alexander Panchenko, and Chris Biemann. 2019. Improving neural entity disambiguation with graph embeddings. In Proceedings of the 57th annual meeting of the association for computational linguistics: student research workshop. 315\u2013322."},{"key":"e_1_2_1_45_1","volume-title":"Generative multimodal entity linking. arXiv preprint arXiv:2306.12725","author":"Shi Senbao","year":"2023","unstructured":"Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. 2023. Generative multimodal entity linking. arXiv preprint arXiv:2306.12725 (2023)."},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"38","author":"Song Shezheng","year":"2024","unstructured":"Shezheng Song, Shan Zhao, Chengyu Wang, Tianwei Yan, Shasha Li, Xiaoguang Mao, and Meng Wang. 2024. A dual-way enhanced frame work from text matching point of view for multimodal entity linking. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19008\u201319016."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407828"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/321879.321884"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2017.03.017"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bdr.2020.100159"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531867"},{"key":"e_1_2_1_52_1","volume-title":"Henry Zhu, Sheng Zhang, Chung-Wei Hang, Pramuditha Perera, Jie Ma, William Wang, Zhiguo Wang, Vittorio Castelli, et al.","author":"Wang Sijia","year":"2023","unstructured":"Sijia Wang, Alexander Hanbo Li, Henry Zhu, Sheng Zhang, Chung-Wei Hang, Pramuditha Perera, Jie Ma, William Wang, Zhiguo Wang, Vittorio Castelli, et al. 2023. Benchmarking diverse-modal entity linking with generative models. arXiv preprint arXiv:2305.17337 (2023)."},{"key":"e_1_2_1_53_1","volume-title":"WikiDiverse: a multimodal entity linking dataset with diversified contextual topics and entity types. arXiv preprint arXiv:2204.06347","author":"Wang Xuwu","year":"2022","unstructured":"Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Rui Wang, Ming Yan, Lihan Chen, and Yanghua Xiao. 2022. WikiDiverse: a multimodal entity linking dataset with diversified contextual topics and entity types. arXiv preprint arXiv:2204.06347 (2022)."},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of The Web Conference","author":"Wu Junshuang","year":"2020","unstructured":"Junshuang Wu, Richong Zhang, Yongyi Mao, Hongyu Guo, Masoumeh Soflaei, and Jinpeng Huai. 2020. Dynamic graph convolutional networks for entity linking. In Proceedings of The Web Conference 2020. 1149\u20131159."},{"key":"e_1_2_1_55_1","volume-title":"Scalable zero-shot entity linking with dense entity retrieval. arXiv preprint arXiv:1911.03814","author":"Wu Ledell","year":"2019","unstructured":"Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2019. Scalable zero-shot entity linking with dense entity retrieval. arXiv preprint arXiv:1911.03814 (2019)."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3612575"},{"key":"e_1_2_1_57_1","unstructured":"Yuan Yao Tianyu Yu Ao Zhang Chongyi Wang Junbo Cui Hongji Zhu Tianchi Cai Haoyu Li Weilin Zhao Zhihui He et al. 2024. MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv preprint arXiv:2408.01800 (2024)."},{"key":"e_1_2_1_58_1","volume-title":"International Conference on Database Systems for Advanced Applications. Springer, 607\u2013622","author":"Zhang Gongrui","year":"2023","unstructured":"Gongrui Zhang, Chenghuan Jiang, Zhongheng Guan, and Peng Wang. 2023. Multimodal entity linking with mixed fusion mechanism. In International Conference on Database Systems for Advanced Applications. Springer, 607\u2013622."},{"key":"e_1_2_1_59_1","volume-title":"Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11\u201314, 2021, Proceedings, Part II 26","author":"Zhang Li","year":"2021","unstructured":"Li Zhang, Zhixu Li, and Qiang Yang. 2021. Attention-based multimodal entity linking with high-quality images. In Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11\u201314, 2021, Proceedings, Part II 26. Springer, 533\u2013548."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3742728.3742740","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T13:34:20Z","timestamp":1756906460000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3742728.3742740"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4]]},"references-count":59,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["10.14778\/3742728.3742740"],"URL":"https:\/\/doi.org\/10.14778\/3742728.3742740","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,4]]},"assertion":[{"value":"2025-09-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}