{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T12:14:11Z","timestamp":1773922451480,"version":"3.50.1"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2019,6,20]],"date-time":"2019-06-20T00:00:00Z","timestamp":1560988800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["5U41 HG006623-02"],"award-info":[{"award-number":["5U41 HG006623-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Automatic biomedical named entity recognition (BioNER) is a key task in biomedical information extraction. For some time, state-of-the-art BioNER has been dominated by machine learning methods, particularly conditional random fields (CRFs), with a recent focus on deep learning. However, recent work has suggested that the high performance of CRFs for BioNER may not generalize to corpora other than the one it was trained on. In our analysis, we find that a popular deep learning-based approach to BioNER, known as bidirectional long short-term memory network-conditional random field (BiLSTM-CRF), is correspondingly poor at generalizing. To address this, we evaluate three modifications of BiLSTM-CRF for BioNER to improve generalization: improved regularization via variational dropout, transfer learning and multi-task learning.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We measure the effect that each strategy has when training\/testing on the same corpus (\u2018in-corpus\u2019 performance) and when training on one corpus and evaluating on another (\u2018out-of-corpus\u2019 performance), our measure of the model\u2019s ability to generalize. We found that variational dropout improves out-of-corpus performance by an average of 4.62%, transfer learning by 6.48% and multi-task learning by 8.42%. The maximal increase we identified combines multi-task learning and variational dropout, which boosts out-of-corpus performance by 10.75%. Furthermore, we make available a new open-source tool, called Saber that implements our best BioNER models.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Source code for our biomedical IE tool is available at https:\/\/github.com\/BaderLab\/saber. Corpora and other resources used in this study are available at https:\/\/github.com\/BaderLab\/Towards-reliable-BioNER.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btz504","type":"journal-article","created":{"date-parts":[[2019,6,17]],"date-time":"2019-06-17T15:11:28Z","timestamp":1560784288000},"page":"280-286","source":"Crossref","is-referenced-by-count":65,"title":["Towards reliable named entity recognition in the biomedical domain"],"prefix":"10.1093","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9621-5046","authenticated-orcid":false,"given":"John M","family":"Giorgi","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Toronto , Toronto, ON M5S 3G4, Canada"},{"name":"The Donnelly Centre, University of Toronto , Toronto, ON M5S 3E1, Canada"}]},{"given":"Gary D","family":"Bader","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Toronto , Toronto, ON M5S 3G4, Canada"},{"name":"The Donnelly Centre, University of Toronto , Toronto, ON M5S 3E1, Canada"},{"name":"Department of Molecular Genetics, University of Toronto , Toronto, ON M5S 1A8, Canada"}]}],"member":"286","published-online":{"date-parts":[[2019,6,20]]},"reference":[{"key":"2023013109504904400_btz504-B1","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1613\/jair.731","article-title":"A model of inductive bias learning","volume":"12","author":"Baxter","year":"2000","journal-title":"J. Artif. Intell. Res"},{"key":"2023013109504904400_btz504-B2","author":"Bayer","year":"2013"},{"key":"2023013109504904400_btz504-B3","doi-asserted-by":"crossref","first-page":"54.","DOI":"10.1186\/1471-2105-14-54","article-title":"Gimli: open source and high-performance biomedical name recognition","volume":"14","author":"Campos","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023013109504904400_btz504-B4","doi-asserted-by":"crossref","first-page":"281.","DOI":"10.1186\/1471-2105-14-281","article-title":"A modular framework for biomedical concept recognition","volume":"14","author":"Campos","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023013109504904400_btz504-B5","first-page":"41","author":"Caruana","year":"1993"},{"key":"2023013109504904400_btz504-B6","first-page":"41","volume-title":"Mach. Learn","author":"Caruana","year":"1997"},{"key":"2023013109504904400_btz504-B7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.0040020","article-title":"Getting started in text mining","volume":"4","author":"Cohen","year":"2008","journal-title":"PLoS Comput. Biol"},{"key":"2023013109504904400_btz504-B8","doi-asserted-by":"crossref","first-page":"368.","DOI":"10.1186\/s12859-017-1776-8","article-title":"A neural network multi-task learning approach to biomedical named entity recognition","volume":"18","author":"Crichton","year":"2017","journal-title":"BMC Bioinformatics"},{"key":"2023013109504904400_btz504-B9","first-page":"12","author":"Del\u0117ger","year":"2016"},{"key":"2023013109504904400_btz504-B10","author":"Devlin","year":"2018"},{"key":"2023013109504904400_btz504-B11","first-page":"1019","volume-title":"Advances in Neural Information Processing Systems","author":"Gal","year":"2016"},{"key":"2023013109504904400_btz504-B12","doi-asserted-by":"crossref","first-page":"2474","DOI":"10.1093\/bioinformatics\/bty152","article-title":"Exploiting and assessing multi-source data for supervised biomedical named entity recognition","volume":"34","author":"Galea","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013109504904400_btz504-B13","doi-asserted-by":"crossref","first-page":"4087","DOI":"10.1093\/bioinformatics\/bty449","article-title":"Transfer learning for biomedical named entity recognition with neural networks","volume":"34","author":"Giorgi","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013109504904400_btz504-B14","doi-asserted-by":"crossref","first-page":"i37","DOI":"10.1093\/bioinformatics\/btx228","article-title":"Deep learning with word embeddings improves biomedical named entity recognition","volume":"33","author":"Habibi","year":"2017","journal-title":"Bioinformatics"},{"key":"2023013109504904400_btz504-B15","author":"Howard","year":"2018"},{"key":"2023013109504904400_btz504-B16","author":"Huang","year":"2015"},{"key":"2023013109504904400_btz504-B17","first-page":"2923","author":"Kafkas","year":"2012"},{"key":"2023013109504904400_btz504-B18","first-page":"70","author":"Kim","year":"2004"},{"key":"2023013109504904400_btz504-B19","author":"Lample","year":"2016"},{"key":"2023013109504904400_btz504-B20","author":"Lee","year":"2019"},{"key":"2023013109504904400_btz504-B21","first-page":"8","author":"Li","year":"2012"},{"key":"2023013109504904400_btz504-B22","author":"Ma","year":"2016"},{"key":"2023013109504904400_btz504-B23","author":"Pachitariu","year":"2013"},{"key":"2023013109504904400_btz504-B24","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng"},{"key":"2023013109504904400_btz504-B25","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1142\/S0219720010004562","article-title":"CALBC silver standard corpus","volume":"8","author":"Rebholz-Schuhmann","year":"2010","journal-title":"J. Bioinform. Comput. Biol"},{"key":"2023013109504904400_btz504-B26","author":"Reimers","year":"2017"},{"key":"2023013109504904400_btz504-B27","doi-asserted-by":"crossref","first-page":"e1000411.","DOI":"10.1371\/journal.pcbi.1000411","article-title":"Getting started in text mining: part two","volume":"5","author":"Rzhetsky","year":"2009","journal-title":"PLoS Comput. Biol"},{"key":"2023013109504904400_btz504-B28","first-page":"383","author":"Sachan","year":"2018"},{"key":"2023013109504904400_btz504-B29","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J. Mach. Learn. Res"},{"key":"2023013109504904400_btz504-B30","first-page":"640","author":"Thrun","year":"1996"},{"key":"2023013109504904400_btz504-B31","first-page":"1745","author":"Wang","year":"2018"},{"key":"2023013109504904400_btz504-B32","doi-asserted-by":"crossref","DOI":"10.1186\/s40537-016-0043-6","article-title":"A survey of transfer learning","volume":"3","author":"Weiss","year":"2016","journal-title":"J. Big Data"},{"key":"2023013109504904400_btz504-B33","author":"Zaremba","year":"2014"},{"key":"2023013109504904400_btz504-B34","doi-asserted-by":"crossref","first-page":"1178","DOI":"10.1093\/bioinformatics\/bth060","article-title":"Recognizing names in biomedical texts: a machine learning approach","volume":"20","author":"Zhou","year":"2004","journal-title":"Bioinformatics"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btz504\/28934727\/btz504.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/1\/280\/48981640\/bioinformatics_36_1_280.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/1\/280\/48981640\/bioinformatics_36_1_280.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T13:32:59Z","timestamp":1675171979000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/36\/1\/280\/5520946"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2019,6,20]]},"references-count":34,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btz504","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/526244","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,1,1]]},"published":{"date-parts":[[2019,6,20]]}}}