{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T23:20:56Z","timestamp":1778887256553,"version":"3.51.4"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"1","content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2006,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Background<\/jats:title>\n            <jats:p>Text mining in the biomedical domain is receiving increasing attention. A key component of this process is named entity recognition (NER). Generally speaking, two annotated corpora, GENIA and GENETAG, are most frequently used for training and testing biomedical named entity recognition (Bio-NER) systems. JNLPBA and BioCreAtIvE are two major Bio-NER tasks using these corpora. Both tasks take different approaches to corpus annotation and use different matching criteria to evaluate system performance. This paper details these differences and describes alternative criteria. We then examine the impact of different criteria and annotation schemes on system performance by retesting systems participated in the above two tasks.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>To analyze the difference between JNLPBA's and BioCreAtIvE's evaluation, we conduct Experiment 1 to evaluate the top four JNLPBA systems using BioCreAtIvE's classification scheme. We then compare them with the top four BioCreAtIvE systems. Among them, three systems participated in both tasks, and each has an F-score lower on JNLPBA than on BioCreAtIvE. In Experiment 2, we apply hypothesis testing and correlation coefficient to find alternatives to BioCreAtIvE's evaluation scheme. It shows that right-match and left-match criteria have no significant difference with BioCreAtIvE. In Experiment 3, we propose a customized relaxed-match criterion that uses right match and merges JNLPBA's five NE classes into two, which achieves an F-score of 81.5%. In Experiment 4, we evaluate a range of five matching criteria from loose to strict on the top JNLPBA system and examine the percentage of false negatives. Our experiment gives the relative change in precision, recall and F-score as matching criteria are relaxed.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>In many applications, biomedical NEs could have several acceptable tags, which might just differ in their left or right boundaries. However, most corpora annotate only one of them. In our experiment, we found that right match and left match can be appropriate alternatives to JNLPBA and BioCreAtIvE's matching criteria. In addition, our relaxed-match criterion demonstrates that users can define their own relaxed criteria that correspond more realistically to their application requirements.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1186\/1471-2105-7-92","type":"journal-article","created":{"date-parts":[[2006,2,24]],"date-time":"2006-02-24T19:14:30Z","timestamp":1140808470000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":67,"title":["Various criteria in the evaluation of biomedical named entity recognition"],"prefix":"10.1186","volume":"7","author":[{"given":"Richard Tzong-Han","family":"Tsai","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shih-Hung","family":"Wu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wen-Chi","family":"Chou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yu-Chun","family":"Lin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ding","family":"He","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jieh","family":"Hsiang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ting-Yi","family":"Sung","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wen-Lian","family":"Hsu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2006,2,24]]},"reference":[{"key":"831_CR1","doi-asserted-by":"publisher","DOI":"10.3115\/1218955.1219010","volume-title":"Classifying Semantic Relations in Bioscience Text","author":"B Rosario","year":"2004","unstructured":"Rosario B, Hearst M: Classifying Semantic Relations in Bioscience Text. 2004."},{"key":"831_CR2","volume-title":"Text Detective: BioAlma's gene annotation tool","author":"J Tamames","year":"2004","unstructured":"Tamames J: Text Detective: BioAlma's gene annotation tool. 2004."},{"key":"831_CR3","volume-title":"Unsupervised Learning of Semantic Relations between Concepts of a Molecular Biology Ontology","author":"M Ciaramita","year":"2005","unstructured":"Ciaramita M, Gangemi A, Ratsch E, Saric J, Rojas I: Unsupervised Learning of Semantic Relations between Concepts of a Molecular Biology Ontology. 2005."},{"issue":"11","key":"831_CR4","doi-asserted-by":"publisher","first-page":"1417","DOI":"10.1093\/bioinformatics\/btg160","volume":"19","author":"JH Chiang","year":"2003","unstructured":"Chiang JH, Yu HC: MeKE: Discovering the Functions of Gene Products from Biomedical Literature via Sentence Alignment. Bioinformatics 2003, 19(11):1417\u20131422. 10.1093\/bioinformatics\/btg160","journal-title":"Bioinformatics"},{"key":"831_CR5","series-title":"Bioinformatics","volume-title":"GENIA corpus - a semantically annotated corpus for bio-textmining","author":"JD Kim","year":"2003","unstructured":"Kim JD, Ohta T, Teteisi Y, Tsujii J: GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics 2003., 19(suppl. 1):"},{"key":"831_CR6","series-title":"BMC Bioinformatics","volume-title":"BioCreAtIvE Task 1A: gene mention finding evaluation","author":"A Yeh","year":"2005","unstructured":"Yeh A, Morgan A, Colosimo M, Hirschman L: BioCreAtIvE Task 1A: gene mention finding evaluation. BMC Bioinformatics 2005., 6(Suppl 1):"},{"key":"831_CR7","series-title":"BMC Bioinformatics","volume-title":"GENETAG: a tagged corpus for gene\/protein named entity recognition","author":"L Tanabe","year":"2005","unstructured":"Tanabe L, Xie N, Thom LH, Matten W, Wilbur WJ: GENETAG: a tagged corpus for gene\/protein named entity recognition. BMC Bioinformatics 2005., 6(Suppl 1):"},{"issue":"5-6","key":"831_CR8","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1016\/j.compbiolchem.2004.09.010","volume":"28","author":"ZZ Hua","year":"2004","unstructured":"Hua ZZ, Mani I, Hermosoa V, Liuc H, Wu CH: iProLINK: an integrated protein resource for literature mining. Computational Biology and Chemistry 2004, 28 (5\u20136):409\u2013416. 10.1016\/j.compbiolchem.2004.09.010","journal-title":"Computational Biology and Chemistry"},{"key":"831_CR9","series-title":"Bioinformatics","volume-title":"Tagging gene and protein names in biomedical text","author":"L Tanabe","year":"2002","unstructured":"Tanabe L, Wilbur WJ: Tagging gene and protein names in biomedical text. Bioinformatics 2002., 18(8):"},{"key":"831_CR10","volume-title":"Using biological resources to bootstrap text mining","author":"L Hirschman","year":"2003","unstructured":"Hirschman L: Using biological resources to bootstrap text mining. 2003."},{"key":"831_CR11","volume-title":"Corpus resources for development and evaluation of a biological text mining system","author":"G Demetrious","year":"2003","unstructured":"Demetrious G, Gaizauskas R: Corpus resources for development and evaluation of a biological text mining system. 2003."},{"key":"831_CR12","doi-asserted-by":"publisher","DOI":"10.1109\/CSB.2003.1227325","volume-title":"A Probabilistic Model for Identifying Protein Names and their Name Boundaries","author":"K Seki","year":"2003","unstructured":"Seki K, Mostafa J: A Probabilistic Model for Identifying Protein Names and their Name Boundaries. 2003."},{"key":"831_CR13","unstructured":"Fukuda KI: KEX download site.[http:\/\/www.hgc.jp\/service\/tooldoc\/KeX\/intro.html]"},{"key":"831_CR14","doi-asserted-by":"publisher","DOI":"10.3115\/1072228.1072338","volume-title":"Notions of correctness when evaluating protein name taggers","author":"F Olsson","year":"2002","unstructured":"Olsson F, Eriksson G, Franz\u00b4en K, Asker L, Lid\u00b4en P: Notions of correctness when evaluating protein name taggers. 2002."},{"key":"831_CR15","volume-title":"Toward information extraction: identifying protein names from biological papers","author":"K Fukuda","year":"1998","unstructured":"Fukuda K, Tsunoda T, Tamura A, Takagi T: Toward information extraction: identifying protein names from biological papers. 1998."},{"key":"831_CR16","volume-title":"Introduction to the Bio-Entity Task at JNLPBA","author":"JD Kim","year":"2004","unstructured":"Kim JD, Ohta T, Tsuruoka Y, Tateisi Y, Collier N: Introduction to the Bio-Entity Task at JNLPBA. 2004."},{"key":"831_CR17","doi-asserted-by":"publisher","DOI":"10.3115\/1567594.1567614","volume-title":"Exploiting Context for Biomedical Entity Recognition: From Syntax to the Web","author":"J Finkel","year":"2004","unstructured":"Finkel J, Dingare S, Nguyen H, Nissim M, Manning C, Sinclair G: Exploiting Context for Biomedical Entity Recognition: From Syntax to the Web. 2004."},{"key":"831_CR18","doi-asserted-by":"crossref","unstructured":"Blaschke C, Leon EA, Krallinger M, Valencia A: Evaluation of BioCreAtIvE assessment of task 2. BMC Bioinformatics 2005., 6(Suppl 1):","DOI":"10.1186\/1471-2105-6-S1-S16"},{"key":"831_CR19","doi-asserted-by":"publisher","DOI":"10.1145\/956863.956941","volume-title":"Information Extraction from Biomedical Literature: Methodology, Evaluation and an Application","author":"LV Subramaniam","year":"2003","unstructured":"Subramaniam LV, Mukherjea S, Kankar P, Srivastava B, Batra VS, Kamesam PV, Kothari R: Information Extraction from Biomedical Literature: Methodology, Evaluation and an Application. 2003."},{"issue":"Suppl. 1","key":"831_CR20","doi-asserted-by":"publisher","first-page":"S97","DOI":"10.1093\/bioinformatics\/17.suppl_1.S97","volume":"17","author":"V Hatzivassiloglous","year":"2001","unstructured":"Hatzivassiloglous V, Duboue P, Rzhetsky A: Disambiguating proteins, genes, and RNA in text: a machine learning approach. Bioinformatics 2001, 17(Suppl. 1):S97-S106.","journal-title":"Bioinformatics"},{"key":"831_CR21","volume-title":"Recognizing Names in Biomedical Texts using Hidden Markov and SVM plus Sigmoid","author":"GD Zhou","year":"2004","unstructured":"Zhou GD: Recognizing Names in Biomedical Texts using Hidden Markov and SVM plus Sigmoid. 2004."},{"key":"831_CR22","doi-asserted-by":"publisher","DOI":"10.3115\/1567594.1567618","volume-title":"Biomedical Named Entity Recognition Using Conditional Random Fields and Rich Feature Sets","author":"B Settles","year":"2004","unstructured":"Settles B: Biomedical Named Entity Recognition Using Conditional Random Fields and Rich Feature Sets. 2004."},{"key":"831_CR23","doi-asserted-by":"publisher","DOI":"10.3115\/1567594.1567617","volume-title":"POSBIOTM-NER in the shared task of BioNLP\/NLPBA 2004: Geneva, Switzerland.","author":"Y Song","year":"2004","unstructured":"Song Y, Kim E, Lee GG, Yi B: POSBIOTM-NER in the shared task of BioNLP\/NLPBA 2004: Geneva, Switzerland. ; 2004."},{"key":"831_CR24","series-title":"BMC Bioinformatics","volume-title":"Recognition of protein\/gene names from text using an ensemble of classifiers","author":"GD Zhou","year":"2005","unstructured":"Zhou GD, Shen D, Zhang J, Su J, Tan SH: Recognition of protein\/gene names from text using an ensemble of classifiers. BMC Bioinformatics 2005., 6(Suppl 1):"},{"key":"831_CR25","series-title":"BMC Bioinformatics","volume-title":"Exploring the boundaries: gene and protein identification in biomedical text","author":"J Finkel","year":"2005","unstructured":"Finkel J, Dingare S, Manning CD, Nissim M, Alex B, Grover C: Exploring the boundaries: gene and protein identification in biomedical text. BMC Bioinformatics 2005., 6(Suppl 1):"},{"key":"831_CR26","series-title":"BMC Bioinformatics","volume-title":"Identifying gene and protein mentions in text using conditional random fields","author":"R McDonald","year":"2005","unstructured":"McDonald R, Pereira F: Identifying gene and protein mentions in text using conditional random fields. BMC Bioinformatics 2005., 6(Suppl 1):"},{"key":"831_CR27","volume-title":"POSBIOTMNER: A Machine Learning Approach for BioNamed Entity Recognition.","author":"Y Song","year":"2004","unstructured":"Song Y, Yi E, Kim E, Lee GG: POSBIOTMNER: A Machine Learning Approach for BioNamed Entity Recognition. 2004."},{"key":"831_CR28","unstructured":"EverythingBio.com: EverythingBio.[http:\/\/www.everythingbio.com\/glos\/definition.php?word=macromolecule]"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/1471-2105-7-92.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,1]],"date-time":"2021-09-01T03:16:48Z","timestamp":1630466208000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/1471-2105-7-92"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2006,2,24]]},"references-count":28,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2006,12]]}},"alternative-id":["831"],"URL":"https:\/\/doi.org\/10.1186\/1471-2105-7-92","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2006,2,24]]},"assertion":[{"value":"30 September 2005","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 February 2006","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 February 2006","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"92"}}