{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T06:27:51Z","timestamp":1774074471228,"version":"3.50.1"},"reference-count":59,"publisher":"Oxford University Press (OUP)","issue":"23","license":[{"start":{"date-parts":[[2018,6,1]],"date-time":"2018-06-01T00:00:00Z","timestamp":1527811200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"US National Institutes of Health","doi-asserted-by":"crossref","award":["#5U41 HG006623-02"],"award-info":[{"award-number":["#5U41 HG006623-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>The explosive increase of biomedical literature has made information extraction an increasingly important tool for biomedical research. A fundamental task is the recognition of biomedical named entities in text (BNER) such as genes\/proteins, diseases and species. Recently, a domain-independent method based on deep learning and statistical word embeddings, called long short-term memory network-conditional random field (LSTM-CRF), has been shown to outperform state-of-the-art entity-specific BNER tools. However, this method is dependent on gold-standard corpora (GSCs) consisting of hand-labeled entities, which tend to be small but highly reliable. An alternative to GSCs are silver-standard corpora (SSCs), which are generated by harmonizing the annotations made by several automatic annotation systems. SSCs typically contain more noise than GSCs but have the advantage of containing many more training examples. Ideally, these corpora could be combined to achieve the benefits of both, which is an opportunity for transfer learning. In this work, we analyze to what extent transfer learning improves upon state-of-the-art results for BNER.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We demonstrate that transferring a deep neural network (DNN) trained on a large, noisy SSC to a smaller, but more reliable GSC significantly improves upon state-of-the-art results for BNER. Compared to a state-of-the-art baseline evaluated on 23 GSCs covering four different entity classes, transfer learning results in an average reduction in error of approximately 11%. We found transfer learning to be especially beneficial for target datasets with a small number of labels (approximately 6000 or less).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Source code for the LSTM-CRF is available at https:\/\/github.com\/Franck-Dernoncourt\/NeuroNER\/ and links to the corpora are available at https:\/\/github.com\/BaderLab\/Transfer-Learning-BNER-Bioinformatics-2018\/.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/bty449","type":"journal-article","created":{"date-parts":[[2018,5,29]],"date-time":"2018-05-29T15:12:41Z","timestamp":1527606761000},"page":"4087-4094","source":"Crossref","is-referenced-by-count":134,"title":["Transfer learning for biomedical named entity recognition with neural networks"],"prefix":"10.1093","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9621-5046","authenticated-orcid":false,"given":"John M","family":"Giorgi","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Toronto, Toronto, Canada"},{"name":"The Donnelly Centre, University of Toronto, Toronto, Canada"}]},{"given":"Gary D","family":"Bader","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Toronto, Toronto, Canada"},{"name":"The Donnelly Centre, University of Toronto, Toronto, Canada"},{"name":"Department of Molecular Genetics, University of Toronto, Toronto, ON, Canada"}]}],"member":"286","published-online":{"date-parts":[[2018,6,1]]},"reference":[{"key":"2023012712301449400_bty449-B1","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1038\/nbt1203","article-title":"Gene prioritization through genomic data fusion","volume":"24","author":"Aerts","year":"2006","journal-title":"Nat. Biotechnol."},{"key":"2023012712301449400_bty449-B2","doi-asserted-by":"crossref","first-page":"e107477","DOI":"10.1371\/journal.pone.0107477","article-title":"Annotated chemical patent corpus: a gold standard for text mining","volume":"9","author":"Akhondi","year":"2014","journal-title":"PLoS One"},{"key":"2023012712301449400_bty449-B3","doi-asserted-by":"crossref","first-page":"15784","DOI":"10.1038\/s41598-017-16081-2","article-title":"Constructing genetic networks using biomedical literature and rare event classification","volume":"7","author":"Al-Aamri","year":"2017","journal-title":"Sci. Rep."},{"key":"2023012712301449400_bty449-B4","doi-asserted-by":"crossref","DOI":"10.12688\/f1000research.4591.2","article-title":"Detecting miRNA mentions and relations in biomedical literature","volume":"3","author":"Bagewadi","year":"2014","journal-title":"F1000Research"},{"key":"2023012712301449400_bty449-B5","first-page":"3","article-title":"A model of inductive bias learning","volume":"12","author":"Baxter","year":"2000","journal-title":"J. Artif. Intell. Res. (JAIR)"},{"key":"2023012712301449400_bty449-B6","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1109\/72.279181","article-title":"Learning long-term dependencies with gradient descent is difficult","volume":"5","author":"Bengio","year":"1994","journal-title":"IEEE Trans. Neural Netw."},{"key":"2023012712301449400_bty449-B7","doi-asserted-by":"crossref","DOI":"10.5772\/51066","article-title":"Biomedical named entity recognition: a survey of machine-learning tools","volume-title":"Theory and Applications for Advanced Text Mining","author":"Campos","year":"2012"},{"key":"2023012712301449400_bty449-B8","first-page":"41","article-title":"Multitask learning: A knowledge-based source of inductive bias","volume-title":"Proceedings of the Tenth International Conference on Machine Learning","author":"Caruana","year":"1993"},{"key":"2023012712301449400_bty449-B9","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/978-1-4615-5529-2_5","article-title":"Multitask learning","volume-title":"Learning to Learn","author":"Caruana","year":"1998"},{"key":"2023012712301449400_bty449-B10","first-page":"101","article-title":"Assessing the practical usability of an automatically annotated corpus","volume-title":"Proceedings of the 5th Linguistic Annotation Workshop","author":"Chowdhury","year":"2011"},{"key":"2023012712301449400_bty449-B11","first-page":"2493","article-title":"Natural language processing (almost) from scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"2023012712301449400_bty449-B12","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1186\/s12859-017-1776-8","article-title":"A neural network multi-task learning approach to biomedical named entity recognition","volume":"18","author":"Crichton","year":"2017","journal-title":"BMC Bioinformatics"},{"key":"2023012712301449400_bty449-B13","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1093\/jamia\/ocw156","article-title":"De-identification of patient notes with recurrent neural networks","volume":"24","author":"Dernoncourt","year":"2017","journal-title":"J. Am. Med. Inf. Assoc."},{"key":"2023012712301449400_bty449-B14","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D17-2017","article-title":"NeuroNER: an easy-to-use program for named-entity recognition based on neural networks","volume-title":"Conference on Empirical Methods on Natural Language Processing (EMNLP)","author":"Dernoncourt","year":"2017"},{"key":"2023012712301449400_bty449-B15","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1142\/9789812799623_0031","article-title":"Mining medline: abstracts, sentences, or phrases?","volume-title":"Biocomputing 2002","author":"Ding","year":"2001"},{"key":"2023012712301449400_bty449-B16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Do\u011fan","year":"2014","journal-title":"J. Biomed. Inf."},{"key":"2023012712301449400_bty449-B17","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/1471-2105-11-85","article-title":"Linnaeus: a species name identification system for biomedical literature","volume":"11","author":"Gerner","year":"2010","journal-title":"BMC Bioinformatics"},{"key":"2023012712301449400_bty449-B18","doi-asserted-by":"crossref","first-page":"A4","DOI":"10.1186\/1753-6561-9-S5-A4","article-title":"Linked annotations: a middle ground for manual curation of biomedical databases and text corpora","volume":"9","author":"Goldberg","year":"2015","journal-title":"BMC Proceedings"},{"key":"2023012712301449400_bty449-B19","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional lstm and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Netw."},{"key":"2023012712301449400_bty449-B20","doi-asserted-by":"crossref","first-page":"i37","DOI":"10.1093\/bioinformatics\/btx228","article-title":"Deep learning with word embeddings improves biomedical named entity recognition","volume":"33","author":"Habibi","year":"2017","journal-title":"Bioinformatics"},{"key":"2023012712301449400_bty449-B21","article-title":"A proposal for a configurable silver standard","volume-title":"Proceedings of the Fourth Linguistic Annotation Workshop","author":"Hahn","year":"2010"},{"key":"2023012712301449400_bty449-B22","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"2023012712301449400_bty449-B23","volume-title":"binr: Cut Numeric Values into Evenly Distributed Groups","author":"Izrailev","year":"2015"},{"key":"2023012712301449400_bty449-B24","first-page":"2923","article-title":"Calbc: releasing the final corpora","volume-title":"LREC","author":"Kafkas","year":"2012"},{"key":"2023012712301449400_bty449-B25","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/1758-2946-7-S1-S2","article-title":"The chemdner corpus of chemicals and drugs and its annotation principles","volume":"7","author":"Krallinger","year":"2015","journal-title":"J. Cheminf."},{"key":"2023012712301449400_bty449-B26","first-page":"63","article-title":"Overview of the chemdner patents task","volume-title":"Proceedings of the Fifth BioCreative Challenge Evaluation Workshop","author":"Krallinger","year":"2015"},{"key":"2023012712301449400_bty449-B27","article-title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","volume-title":"Proceedings of the Eighteenth International Conference on Machine Learning","author":"Lafferty","year":"2001"},{"key":"2023012712301449400_bty449-B28","first-page":"260","article-title":"Neural architectures for named entity recognition","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Lample","year":"2016"},{"key":"2023012712301449400_bty449-B29","article-title":"Enabling recognition of diseases in biomedical text with machine learning: corpus and benchmark","volume-title":"Proceedings of the 2009 Symposium on Languages in Biology and Medicine","author":"Leaman","year":"2009"},{"key":"2023012712301449400_bty449-B30","first-page":"abs\/1705.06273","article-title":"Transfer learning for named-entity recognition with neural networks","author":"Lee","year":"2017","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B31","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1093\/bib\/6.4.357","article-title":"What makes a gene name? Named entity recognition in the biomedical literature","volume":"6","author":"Leser","year":"2005","journal-title":"Brief. Bioinf."},{"key":"2023012712301449400_bty449-B32","doi-asserted-by":"crossref","first-page":"e1004391","DOI":"10.1371\/journal.pcbi.1004391","article-title":"miRTex: a text mining system for miRNA-gene relation extraction","volume":"11","author":"Li","year":"2015","journal-title":"PLoS Comput. Biol."},{"key":"2023012712301449400_bty449-B33","doi-asserted-by":"crossref","first-page":"baw068","DOI":"10.1093\/database\/baw068","article-title":"Biocreative v cdr task corpus: a resource for chemical disease relation extraction","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"key":"2023012712301449400_bty449-B34","first-page":"8","volume-title":"Literature Survey: Domain Adaptation Algorithms for Natural Language Processing","author":"Li","year":"2012"},{"key":"2023012712301449400_bty449-B35","article-title":"Finding function in form: compositional character models for open vocabulary word representation","author":"Ling","year":"2015","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B36","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","volume-title":"Advances in Neural Information Processing Systems","author":"Mikolov","year":"2013"},{"key":"2023012712301449400_bty449-B37","article-title":"How transferable are neural networks in NLP applications?","author":"Mou","year":"2016","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B38","first-page":"16","article-title":"Annotating and evaluating text for stem cell research","volume-title":"Proceedings of the Third Workshop on Building and Evaluation Resources for Biomedical Text Mining (BioTxtM 2012) at Language Resources and Evaluation (LREC). Istanbul, Turkey","author":"Neves","year":"2012"},{"key":"2023012712301449400_bty449-B39","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2014.222","article-title":"Learning and transferring mid-level image representations using convolutional neural networks","volume-title":"Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition","author":"Oquab","year":"2014"},{"key":"2023012712301449400_bty449-B40","doi-asserted-by":"crossref","first-page":"e65390","DOI":"10.1371\/journal.pone.0065390","article-title":"The SPECIES and ORGANISMS resources for fast and accurate identification of taxonomic names in text","volume":"8","author":"Pafilis","year":"2013","journal-title":"PLoS One"},{"key":"2023012712301449400_bty449-B41","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"2023012712301449400_bty449-B42","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"Glove: global vectors for word representation","volume-title":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","author":"Pennington","year":"2014"},{"key":"2023012712301449400_bty449-B43","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1186\/1471-2105-8-50","article-title":"Bioinfer: a corpus for information extraction in the biomedical domain","volume":"8","author":"Pyysalo","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023012712301449400_bty449-B44","article-title":"Distributional semantics resources for biomedical text processing","volume-title":"Proceedings of the 5th International Symposium on Languages in Biology and Medicine","author":"Pyysalo","year":"2013"},{"key":"2023012712301449400_bty449-B45","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1038\/nbt.3183","article-title":"Opportunities for drug repositioning from phenome-wide association studies","volume":"33","author":"Rastegar-Mojarad","year":"2015","journal-title":"Nat. Biotechnol."},{"key":"2023012712301449400_bty449-B46","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1142\/S0219720010004562","article-title":"Calbc silver standard corpus","volume":"08","author":"Rebholz-Schuhmann","year":"2010","journal-title":"J. Bioinf. Comput. Biol."},{"key":"2023012712301449400_bty449-B47","article-title":"Optimal hyperparameters for deep LSTM-networks for sequence labeling tasks","author":"Reimers","year":"2017","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B48","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/gb-2008-9-s2-s2","article-title":"Overview of biocreative ii gene mention recognition","volume":"9","author":"Smith","year":"2008","journal-title":"Genome Biol."},{"key":"2023012712301449400_bty449-B49","article-title":"Transfer learning approach for financial applications","author":"Stamate","year":"2015","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B50","first-page":"640","article-title":"Is learning the n-th thing any easier than learning the first?","volume-title":"Advances in Neural Information Processing Systems","author":"Thrun","year":"1996"},{"key":"2023012712301449400_bty449-B51","doi-asserted-by":"crossref","first-page":"bat019","DOI":"10.1093\/database\/bat019","article-title":"Annotating the biomedical literature for the human variome","volume":"2013","author":"Verspoor","year":"2013","journal-title":"Database"},{"key":"2023012712301449400_bty449-B52","first-page":"1225","article-title":"Transfer learning for speech and language processing","volume-title":"Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2015 Asia-Pacific","author":"Wang","year":"2015"},{"key":"2023012712301449400_bty449-B53","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1093\/bioinformatics\/btq002","article-title":"Disambiguating the species of biomedical named entities using natural language parsers","volume":"26","author":"Wang","year":"2010","journal-title":"Bioinformatics"},{"key":"2023012712301449400_bty449-B54","doi-asserted-by":"crossref","first-page":"1080","DOI":"10.1038\/nbt.2758","article-title":"Rational drug repositioning by medical genetics","volume":"31","author":"Wang","year":"2013","journal-title":"Nat. Biotechnol."},{"key":"2023012712301449400_bty449-B55","first-page":"3","article-title":"A survey of transfer learning","author":"Weiss","year":"2016","journal-title":"J. Big Data"},{"key":"2023012712301449400_bty449-B56","article-title":"How transferable are features in deep neural networks?","author":"Yosinski","year":"2014","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B57","doi-asserted-by":"crossref","DOI":"10.1145\/2834892.2834896","article-title":"Optimizing deep learning hyper-parameters through an evolutionary algorithm","volume-title":"Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments","author":"Young","year":"2015"},{"key":"2023012712301449400_bty449-B58","article-title":"Visualizing and understanding convolutional networks","author":"Zeiler","year":"2013","journal-title":"CoRR"},{"key":"2023012712301449400_bty449-B59","first-page":"5","article-title":"Human symptoms\u2013disease network","author":"Zhou","year":"2014","journal-title":"Nat. Commun."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/34\/23\/4087\/48920731\/bioinformatics_34_23_4087.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/34\/23\/4087\/48920731\/bioinformatics_34_23_4087.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T08:18:03Z","timestamp":1674807483000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/34\/23\/4087\/5026661"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2018,6,1]]},"references-count":59,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2018,12,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bty449","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/262790","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,12,1]]},"published":{"date-parts":[[2018,6,1]]}}}