{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:36:32Z","timestamp":1772908592132,"version":"3.50.1"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T00:00:00Z","timestamp":1726790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>With the exponential growth of the life sciences literature, biomedical text mining (BTM) has become an essential technology for accelerating the extraction of insights from publications. The identification of entities in texts, such as diseases or genes, and their normalization, i.e. grounding them in knowledge base, are crucial steps in any BTM pipeline to enable information aggregation from multiple documents. However, tools for these two steps are rarely applied in the same context in which they were developed. Instead, they are applied \u201cin the wild,\u201d i.e. on application-dependent text collections from moderately to extremely different from those used for training, varying, e.g. in focus, genre or text type. This raises the question whether the reported performance, usually obtained by training and evaluating on different partitions of the same corpus, can be trusted for downstream applications.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Here, we report on the results of a carefully designed cross-corpus benchmark for entity recognition and normalization, where tools were applied systematically to corpora not used during their training. Based on a survey of 28 published systems, we selected five, based on predefined criteria like feature richness and availability, for an in-depth analysis on three publicly available corpora covering four entity types. Our results present a mixed picture and show that cross-corpus performance is significantly lower than the in-corpus performance. HunFlair2, the redesigned and extended successor of the HunFlair tool, showed the best performance on average, being closely followed by PubTator Central. Our results indicate that users of BTM tools should expect a lower performance than the original published one when applying tools in \u201cthe wild\u201d and show that further research is necessary for more robust BTM tools.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>All our models are integrated into the Natural Language Processing (NLP) framework flair: https:\/\/github.com\/flairNLP\/flair. Code to reproduce our results is available at: https:\/\/github.com\/hu-ner\/hunflair2-experiments.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btae564","type":"journal-article","created":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T15:28:44Z","timestamp":1726846124000},"source":"Crossref","is-referenced-by-count":15,"title":["HunFlair2 in a cross-corpus evaluation of biomedical named entity recognition and normalization tools"],"prefix":"10.1093","volume":"40","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2950-2587","authenticated-orcid":false,"given":"Mario","family":"S\u00e4nger","sequence":"first","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin , Berlin 10099,","place":["Germany"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8234-8299","authenticated-orcid":false,"given":"Samuele","family":"Garda","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin , Berlin 10099,","place":["Germany"]}]},{"given":"Xing David","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin , Berlin 10099,","place":["Germany"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2499-472X","authenticated-orcid":false,"given":"Leon","family":"Weber-Genzel","sequence":"additional","affiliation":[{"name":"Center for Information and Language Processing (CIS), Ludwig Maximilian University Munich , M\u00fcnchen 80539,","place":["Germany"]}]},{"given":"Pia","family":"Droop","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin , Berlin 10099,","place":["Germany"]}]},{"given":"Benedikt","family":"Fuchs","sequence":"additional","affiliation":[{"name":"Research Industrial Systems Engineering (RISE) Forschungs-, Entwicklungs- und Gro\u00dfprojektberatung GmbH , Schwechat 2320,","place":["Austria"]}]},{"given":"Alan","family":"Akbik","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin , Berlin 10099,","place":["Germany"]}]},{"given":"Ulf","family":"Leser","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Humboldt-Universit\u00e4t zu Berlin , Berlin 10099,","place":["Germany"]}]}],"member":"286","published-online":{"date-parts":[[2024,9,20]]},"reference":[{"key":"2024100514411835500_btae564-B1","first-page":"376","author":"Arighi","year":"2017"},{"key":"2024100514411835500_btae564-B2","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2024100514411835500_btae564-B3","doi-asserted-by":"publisher","first-page":"D36","DOI":"10.1093\/nar\/gku1055","article-title":"Gene: a gene-centered information resource at NCBI","volume":"43","author":"Brown","year":"2015","journal-title":"Nucleic Acids Res"},{"key":"2024100514411835500_btae564-B4","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1186\/s12859-017-1857-8","article-title":"A method for named entity normalization in biomedical articles: application to diseases and plants","volume":"18","author":"Cho","year":"2017","journal-title":"BMC Bioinformatics"},{"key":"2024100514411835500_btae564-B5","first-page":"73","author":"Collier","year":"2004"},{"key":"2024100514411835500_btae564-B6","doi-asserted-by":"publisher","first-page":"D1257","DOI":"10.1093\/nar\/gkac833","article-title":"Comparative Toxicogenomics Database (CTD): update 2023","volume":"51","author":"Davis","year":"2023","journal-title":"Nucleic Acids Res"},{"key":"2024100514411835500_btae564-B7","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-38721-0","volume-title":"Ontology Matching","author":"Euzenat","year":"2013"},{"key":"2024100514411835500_btae564-B8","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1186\/s12859-023-05350-9","article-title":"An analysis of entity normalization evaluation biases in specialized domains","volume":"24","author":"Ferr\u00e9","year":"2023","journal-title":"BMC Bioinformatics"},{"key":"2024100514411835500_btae564-B9","doi-asserted-by":"publisher","first-page":"104252","DOI":"10.1016\/j.jbi.2022.104252","article-title":"An overview of biomedical entity linking throughout the years","volume":"137","author":"French","year":"2022","journal-title":"J Biomed Inform"},{"key":"2024100514411835500_btae564-B10","first-page":"25792","article-title":"BigBIO: a framework for data-centric biomedical natural language processing","volume":"35","author":"Fries","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024100514411835500_btae564-B11","doi-asserted-by":"publisher","first-page":"2474","DOI":"10.1093\/bioinformatics\/bty152","article-title":"Exploiting and assessing multi-source data for supervised biomedical named entity recognition","volume":"34","author":"Galea","year":"2018","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B12","doi-asserted-by":"publisher","first-page":"btad698","DOI":"10.1093\/bioinformatics\/btad698","article-title":"BELB: a biomedical entity linking benchmark","volume":"39","author":"Garda","year":"2023","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B13","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/1471-2105-11-85","article-title":"LINNAEUS: a species name identification system for biomedical literature","volume":"11","author":"Gerner","year":"2010","journal-title":"BMC Bioinformatics"},{"key":"2024100514411835500_btae564-B14","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1093\/bioinformatics\/btz504","article-title":"Towards reliable named entity recognition in the biomedical domain","volume":"36","author":"Giorgi","year":"2020","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B15","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1186\/1471-2105-9-136","article-title":"Mining phenotypes for gene function prediction","volume":"9","author":"Groth","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2024100514411835500_btae564-B16","first-page":"15","author":"Gurulingappa","year":"2010"},{"key":"2024100514411835500_btae564-B17","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1038\/s41597-021-00875-1","article-title":"NLM-Chem, a new resource for chemical entity recognition in PubMed full text literature","volume":"8","author":"Islamaj","year":"2021","journal-title":"Sci Data"},{"key":"2024100514411835500_btae564-B18","doi-asserted-by":"publisher","first-page":"103779","DOI":"10.1016\/j.jbi.2021.103779","article-title":"NLM-Gene, a richly annotated gold standard dataset for gene entities that addresses ambiguity and multi-species gene recognition","volume":"118","author":"Islamaj","year":"2021","journal-title":"J Biomed Inform"},{"key":"2024100514411835500_btae564-B19","doi-asserted-by":"crossref","first-page":"btae163","DOI":"10.1093\/bioinformatics\/btae163","article-title":"Advancing entity recognition in biomedicine via instruction tuning of large language models","volume":"40","author":"Keloth","year":"2024","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B20","author":"Kol\u00e1rik","year":"2008"},{"key":"2024100514411835500_btae564-B21","doi-asserted-by":"publisher","first-page":"820","DOI":"10.1007\/s10618-014-0382-x","article-title":"Evaluation measures for hierarchical classification: a unified view and novel approaches","volume":"29","author":"Kosmopoulos","year":"2015","journal-title":"Data Min Knowl Disc"},{"key":"2024100514411835500_btae564-B22","doi-asserted-by":"publisher","first-page":"2839","DOI":"10.1093\/bioinformatics\/btw343","article-title":"TaggerOne: joint named entity recognition and normalization with semi-Markov models","volume":"32","author":"Leaman","year":"2016","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B23","doi-asserted-by":"publisher","DOI":"10.1093\/database\/baad005","article-title":"Chemical identification and indexing in full-text articles: an overview of the NLM-Chem track at BioCreative VII","volume":"2023","author":"Leaman","year":"2023","journal-title":"Database"},{"key":"2024100514411835500_btae564-B24","first-page":"4228","author":"Liu","year":"2021"},{"key":"2024100514411835500_btae564-B25","doi-asserted-by":"crossref","first-page":"1275","DOI":"10.1093\/bioinformatics\/btg153","article-title":"Investigating semantic similarity measures across the gene ontology: the relationship between sequence and annotation","volume":"19","author":"Lord","year":"2003","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B26","doi-asserted-by":"crossref","first-page":"bbac282","DOI":"10.1093\/bib\/bbac282","article-title":"BioRED: a rich biomedical relation extraction dataset","volume":"23","author":"Luo","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024100514411835500_btae564-B27","doi-asserted-by":"crossref","first-page":"btad310","DOI":"10.1093\/bioinformatics\/btad310","article-title":"AIONER: all-in-one scheme-based biomedical named entity recognition using deep learning","volume":"39","author":"Luo","year":"2023","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B28","doi-asserted-by":"publisher","author":"Mohan","year":"2019","DOI":"10.24432\/C5G59C"},{"key":"2024100514411835500_btae564-B29","doi-asserted-by":"publisher","first-page":"S3","DOI":"10.1186\/gb-2008-9-s2-s3","article-title":"Overview of BioCreative II gene normalization","volume":"9","author":"Morgan","year":"2008","journal-title":"Genome Biol"},{"key":"2024100514411835500_btae564-B30","doi-asserted-by":"crossref","first-page":"4837","DOI":"10.1093\/bioinformatics\/btac598","article-title":"BERN2: an advanced neural biomedical named entity recognition and normalization tool","volume":"38","author":"Mujeen","year":"2022","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B31","author":"Neumann"},{"key":"2024100514411835500_btae564-B32","doi-asserted-by":"crossref","first-page":"e65390","DOI":"10.1371\/journal.pone.0065390","article-title":"The species and organisms resources for fast and accurate identification of taxonomic names in text","volume":"8","author":"Pafilis","year":"2013","journal-title":"PLoS One"},{"key":"2024100514411835500_btae564-B33","first-page":"58","author":"Pyysalo","year":"2013"},{"key":"2024100514411835500_btae564-B34","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1186\/s13321-020-00461-4","article-title":"Linking chemical and disease entities to ontologies by integrating PageRank with extracted relations from literature","volume":"12","author":"Ruas","year":"2020","journal-title":"J Cheminform"},{"key":"2024100514411835500_btae564-B35","author":"Ruas","year":"2023"},{"key":"2024100514411835500_btae564-B36","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1093\/bioinformatics\/btaa674","article-title":"Large-scale entity representation learning for biomedical relationship extraction","volume":"37","author":"S\u00e4nger","year":"2021","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B37","doi-asserted-by":"crossref","first-page":"D136","DOI":"10.1093\/nar\/gkr1178","article-title":"The NCBI Taxonomy database","volume":"40","author":"Scott","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2024100514411835500_btae564-B38","doi-asserted-by":"crossref","first-page":"bbab282","DOI":"10.1093\/bib\/bbab282","article-title":"Deep learning methods for biomedical named entity recognition: a survey and qualitative comparison","volume":"22","author":"Song","year":"2021","journal-title":"Brief Bioinform"},{"key":"2024100514411835500_btae564-B39","doi-asserted-by":"crossref","first-page":"bbac342","DOI":"10.1093\/bib\/bbac342","article-title":"Deep learning joint models for extracting entities and relations in biomedical: a survey and comparison","volume":"23","author":"Su","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024100514411835500_btae564-B40","first-page":"3641","author":"Sung","year":"2020"},{"key":"2024100514411835500_btae564-B41","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1007\/s13042-015-0426-6","article-title":"A comparative study for biomedical named entity recognition","volume":"9","author":"Wang","year":"2018","journal-title":"Int J Mach Learn Cyber"},{"key":"2024100514411835500_btae564-B42","first-page":"88","author":"Wang","year":"2020"},{"key":"2024100514411835500_btae564-B43","doi-asserted-by":"crossref","first-page":"i490","DOI":"10.1093\/bioinformatics\/btaa430","article-title":"PEDL: extracting protein\u2013protein associations using deep language models and distant supervision","volume":"36","author":"Weber","year":"2020","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B44","doi-asserted-by":"crossref","first-page":"2792","DOI":"10.1093\/bioinformatics\/btab042","article-title":"HunFlair: an easy-to-use tool for state-of-the-art biomedical named entity recognition","volume":"37","author":"Weber","year":"2021","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B45","doi-asserted-by":"crossref","first-page":"baac098","DOI":"10.1093\/database\/baac098","article-title":"Chemical\u2013protein relation extraction with ensembles of carefully tuned pretrained language models","volume":"2022","author":"Weber","year":"2022","journal-title":"Database"},{"key":"2024100514411835500_btae564-B46","doi-asserted-by":"publisher","first-page":"S5","DOI":"10.1186\/1471-2105-12-s8-s5","article-title":"Cross-species gene normalization by species inference","volume":"12","author":"Wei","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2024100514411835500_btae564-B47","doi-asserted-by":"crossref","first-page":"e918710","DOI":"10.1155\/2015\/918710","article-title":"GNormPlus: an integrative approach for tagging genes, gene families, and protein domains","volume":"2015","author":"Wei","year":"2015","journal-title":"Biomed Res Int"},{"key":"2024100514411835500_btae564-B48","doi-asserted-by":"crossref","first-page":"W587","DOI":"10.1093\/nar\/gkz389","article-title":"PubTator Central: automated concept annotation for biomedical full text articles","volume":"47","author":"Wei","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2024100514411835500_btae564-B49","doi-asserted-by":"publisher","first-page":"4449","DOI":"10.1093\/bioinformatics\/btac537","article-title":"tmVar 3.0: an improved variant concept recognition and normalization tool","volume":"38","author":"Wei","year":"2022","journal-title":"Bioinformatics"},{"key":"2024100514411835500_btae564-B50","first-page":"8003","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Yasunaga","year":"2022"},{"key":"2024100514411835500_btae564-B51","doi-asserted-by":"publisher","first-page":"1892","DOI":"10.1093\/jamia\/ocab090","article-title":"Biomedical and clinical English model packages for the Stanza Python NLP library","volume":"28","author":"Zhang","year":"2021","journal-title":"J Am Med Inform Assoc"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae564\/59212710\/btae564.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/10\/btae564\/59604134\/btae564.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/10\/btae564\/59604134\/btae564.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,5]],"date-time":"2024-10-05T14:41:32Z","timestamp":1728139292000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btae564\/7762634"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2024,9,20]]},"references-count":51,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2024,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btae564","relation":{},"ISSN":["1367-4811"],"issn-type":[{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,10]]},"published":{"date-parts":[[2024,9,20]]},"article-number":"btae564"}}