{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,12]],"date-time":"2026-11-12T14:13:57Z","timestamp":1794492837026,"version":"4.9.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T00:00:00Z","timestamp":1770854400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T00:00:00Z","timestamp":1772755200000},"content-version":"vor","delay-in-days":22,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Biomed Semant"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Entity linking maps textual mentions with entities in vocabularies. While accuracy is the primary metric for entity linking evaluation, it fails to capture the complexity of model behaviour.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We propose an entity linking evaluation framework that clarifies the target objects of metrics calculation, incorporates term hierarchy, generates performance profiles, and summarises them as model characteristics. The framework emphasises hierarchical vocabulary structures, shifting the focus from text-level label matching to semantic comparison between concepts. We illustrate the competence and utility of this framework through a case study on disease entity linking.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Our results highlight the importance of aligning evaluation metrics with application-specific requirements, and provide structured hierarchical error analysis for entity linking, paving the way for more nuanced and practical assessments of entity linking systems.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s13326-025-00339-0","type":"journal-article","created":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T09:18:35Z","timestamp":1770887915000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A practical and nuanced framework for entity linking evaluation"],"prefix":"10.1186","volume":"17","author":[{"given":"Fuqi","family":"Xu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Goran","family":"Nenadic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Stevens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,2,12]]},"reference":[{"key":"339_CR1","doi-asserted-by":"crossref","unstructured":"Albertoni R, Camossi E, De Martino M, Giannini F, Monti M. Semantic granularity for the semantic web. In OTM Confederated International Conferences on the Move to Meaningful Internet Systems. Springer; 2006, pp. 1863\u201372.","DOI":"10.1007\/11915072_93"},{"key":"339_CR2","volume-title":"Web ontology language: OWL","author":"G Antoniou","year":"2009","unstructured":"Antoniou G, van Harmelen F. Web ontology language: OWL. Springer; 2009."},{"key":"339_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-7-267","volume":"7","author":"AG Castro","year":"2006","unstructured":"Castro AG, Rocca-Serra P, Stevens R, Taylor C, Nashar K, Ragan MA, Sansone S-A. The use of concept maps during knowledge elicitation in ontology development processes\u2013the nutrigenomics use case. BMC Bioinf. 2006;7:1\u201314.","journal-title":"BMC Bioinf"},{"key":"339_CR4","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1007\/978-3-031-11039-9_17","volume-title":"Terminology, ontology and their implementations","author":"S de Coronado","year":"2023","unstructured":"de Coronado S, Remennik L, Elkin PL. National Cancer Institute thesaurus (NCIt). In Terminology, Ontology and their Implementations. Springer; 2023. p. 395\u2013441."},{"key":"339_CR5","doi-asserted-by":"crossref","unstructured":"French E, McInnes BT. An overview of biomedical entity linking throughout the years. J Biomed Inf. 2023;137(104252).","DOI":"10.1016\/j.jbi.2022.104252"},{"key":"339_CR6","doi-asserted-by":"crossref","unstructured":"Garda S, Leser U. BELHD: improving biomedical entity linking with homonym disambiguation. Bioinformatics, page btae, 2024;474.","DOI":"10.1093\/bioinformatics\/btae474"},{"key":"339_CR7","doi-asserted-by":"crossref","unstructured":"Garda S, Weber-Genzel L, Martin R, Leser U. BELB: a biomedical entity linking benchmark. Bioinformatics. 2023;39(11):btad698.","DOI":"10.1093\/bioinformatics\/btad698"},{"key":"339_CR8","unstructured":"Gupta G, Rastegarpanah B, Iyer A, Rubin J, Kenthapadi K. Measuring distributional shifts in text: the advantage of language model-based embeddings. arXiv preprint arXiv:2312.02337, 2023."},{"key":"339_CR9","unstructured":"Hoehndorf R. Version of MeSH - RH-MESH ontology. http:\/\/phenomebrowser.net\/ontologies\/mesh\/https:\/\/bioportal.bioontology.org\/ontologies\/RH-MESH. Accessed 11 Nov 2024."},{"key":"339_CR10","doi-asserted-by":"crossref","unstructured":"Jin D, Sisman B, Wei H, Dong XL, Koutra D. Deep transfer learning for multi-source entity linkage via domain adaptation. arXiv preprint arXiv:2110.14509, 2021.","DOI":"10.14778\/3494124.3494131"},{"key":"339_CR11","unstructured":"Jonquet C, Shah NH, Youn CH, Musen MA, Callendar C, Storey M-A. NCBO annotator: semantic annotation of biomedical data. In ISWC 2009-8th International Semantic Web Conference, Poster and Demo Session. CEUR Workshop Proceedings. 2009;171."},{"key":"339_CR12","first-page":"101","volume-title":"MEDINFO 2021: One world, one health\u2013global partnership for digital innovation","author":"H Kang","year":"2022","unstructured":"Kang H, Park H. Mapping korean national health insurance reimbursement claim codes for therapeutic and surgical procedures to SNOMED-CT to facilitate data reuse. In MEDINFO 2021: one world, one health\u2013global partnership for digital innovation. IOS Press; 2022. pp. 101\u201305."},{"key":"339_CR13","doi-asserted-by":"crossref","unstructured":"Kartchner D, Deng J, Lohiya S, Kopparthi T, Bathala P, Domingo-Fern\u00e1ndez D, Mitchell CS. A comprehensive evaluation of biomedical entity linking models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. NIH Public Access; 2023, p. 14462.","DOI":"10.18653\/v1\/2023.emnlp-main.893"},{"key":"339_CR14","unstructured":"DMIS Lab. BioSyn: biomedical entity representations with synonym marginalization. GitHub repository. 2020. https:\/\/github.com\/dmis-lab\/BioSyn."},{"key":"339_CR15","doi-asserted-by":"crossref","unstructured":"Leaman R, Islamaj R, Adams V, Alliheedi MA, Rafael Almeida J, Antunes R, Bevan R, Chang Y-C, Erdengasileng A, Hodgskiss M, et al. Chemical identification and indexing in full-text articles: an overview of the nlm-chem track at biocreative vii. Database. 2023, 2023:baad005.","DOI":"10.1093\/database\/baad005"},{"key":"339_CR16","doi-asserted-by":"crossref","unstructured":"Leaman R, Zhiyong L. TaggerOne: joint named entity recognition and normalization with semi-markov models. Bioinformatics. 2016;32(18):2839\u201346.","DOI":"10.1093\/bioinformatics\/btw343"},{"key":"339_CR17","doi-asserted-by":"crossref","unstructured":"Jiao L, Sun Y, Johnson RJ, Sciaky D, Wei C-H, Leaman R, Davis AP, Mattingly CJ, Wiegers TC, Zhiyong L. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database. 2016;2016:baw068.","DOI":"10.1093\/database\/baw068"},{"key":"339_CR18","unstructured":"Carolyn E Lipscomb. Medical subject headings (MeSH). Bulletin Med Library Assoc. 2000;88(3):265."},{"key":"339_CR19","doi-asserted-by":"crossref","unstructured":"Opitz J. A closer look at classification evaluation metrics and a critical reflection of common evaluation practice. Trans Assoc Comput Linguist. 2024;12:820\u201336.","DOI":"10.1162\/tacl_a_00675"},{"key":"339_CR20","doi-asserted-by":"crossref","unstructured":"Rani GJJ, Gladis D, Mammen JJ. SNOMED CT annotation for improved pathological decisions in breast cancer domain. Int J Recent Technol Eng. 2019;8(3):8400\u201306.","DOI":"10.35940\/ijrte.C6519.098319"},{"key":"339_CR21","doi-asserted-by":"crossref","unstructured":"Smith B, Ashburner M, Rosse C, Bard J, Bug W, Ceusters W, Goldberg LJ, Eilbeck K, Ireland A, Mungall CJ, et al. The OBO foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol. 2007;25(11):1251\u201355.","DOI":"10.1038\/nbt1346"},{"key":"339_CR22","doi-asserted-by":"crossref","unstructured":"Sung M, Jeon H, Lee J, Kang J. Biomedical entity representations with synonym marginalization. arXiv preprint arXiv:2005.00239, 2020.","DOI":"10.18653\/v1\/2020.acl-main.335"},{"key":"339_CR23","doi-asserted-by":"crossref","unstructured":"Tutubalina E, Kadurin A, Miftahutdinov Z. Fair evaluation in concept normalization: a large-scale comparative analysis for BERT-Based models. In Proceedings of the 28th International Conference on Computational Linguistics. 2020, pp. 6710\u201316.","DOI":"10.18653\/v1\/2020.coling-main.588"},{"key":"339_CR24","doi-asserted-by":"crossref","unstructured":"Van Erp M, Mendes P, Paulheim H, Ilievski F, Plu J, Rizzo G, Waitelonis J. Evaluating entity linking: an analysis of current benchmark datasets and a roadmap for doing a better job. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916). 2016, pp. 4373\u201379.","DOI":"10.63317\/2hnj3wruemwj"},{"issue":"W1","key":"339_CR25","doi-asserted-by":"publisher","first-page":"W587","DOI":"10.1093\/nar\/gkz389","volume":"47","author":"C-H Wei","year":"2019","unstructured":"Wei C-H, Allot A, Leaman R, Zhiyong L. PubTator central: automated concept annotation for biomedical full text articles. Nucleic Acids Res. 2019;47(W1):W587\u201393.","journal-title":"Nucleic Acids Res"},{"issue":"3","key":"339_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1993036.1993039","volume":"29","author":"X Yan","year":"2011","unstructured":"Yan X, Lau RY, Song D, Xue L, Jian M. Toward a semantic granularity model for domain-specific information retrieval. ACM Trans On Inf Syst (TOIS). 2011;29(3):1\u201346.","journal-title":"ACM Trans On Inf Syst (TOIS)"},{"key":"339_CR27","doi-asserted-by":"crossref","unstructured":"Zhang S, Cheng H, Vashishth S, Wong C, Xiao J, Liu X, Naumann T, Gao J, Poon H. Knowledge-rich self-supervision for biomedical entity linking. arXiv:2112.07887, 2022.","DOI":"10.18653\/v1\/2022.findings-emnlp.61"}],"container-title":["Journal of Biomedical Semantics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13326-025-00339-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-025-00339-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-025-00339-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T00:26:17Z","timestamp":1790036777000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s13326-025-00339-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,12]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["339"],"URL":"https:\/\/doi.org\/10.1186\/s13326-025-00339-0","relation":{},"ISSN":["2041-1480"],"issn-type":[{"value":"2041-1480","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,12]]},"assertion":[{"value":"22 April 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 September 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","label":"Ethics approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"5"}}