{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T09:51:14Z","timestamp":1785577874963,"version":"3.56.0"},"reference-count":65,"publisher":"Oxford University Press (OUP)","issue":"14","license":[{"start":{"date-parts":[[2017,7,12]],"date-time":"2017-07-12T00:00:00Z","timestamp":1499817600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002347","name":"BMBF","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004807","name":"DFG","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004807","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,7,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Text mining has become an important tool for biomedical research. The most fundamental text-mining task is the recognition of biomedical named entities (NER), such as genes, chemicals and diseases. Current NER methods rely on pre-defined features which try to capture the specific surface properties of entity types, properties of the typical local context, background knowledge, and linguistic information. State-of-the-art tools are entity-specific, as dictionaries and empirically optimal feature sets differ between entity types, which makes their development costly. Furthermore, features are often optimized for a specific gold standard corpus, which makes extrapolation of quality measures difficult.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We show that a completely generic method based on deep learning and statistical word embeddings [called long short-term memory network-conditional random field (LSTM-CRF)] outperforms state-of-the-art entity-specific NER tools, and often by a large margin. To this end, we compared the performance of LSTM-CRF on 33 data sets covering five different entity classes with that of best-of-class NER tools and an entity-agnostic CRF implementation. On average, F1-score of LSTM-CRF is 5% above that of the baselines, mostly due to a sharp increase in recall.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>The source code for LSTM-CRF is available at https:\/\/github.com\/glample\/tagger and the links to the corpora are available at https:\/\/corposaurus.github.io\/corpora\/.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btx228","type":"journal-article","created":{"date-parts":[[2017,4,14]],"date-time":"2017-04-14T07:08:51Z","timestamp":1492153731000},"page":"i37-i48","source":"Crossref","is-referenced-by-count":415,"title":["Deep learning with word embeddings improves biomedical named entity recognition"],"prefix":"10.1093","volume":"33","author":[{"given":"Maryam","family":"Habibi","sequence":"first","affiliation":[{"name":"Computer Science Department, Humboldt-Universit\u00e4t zu Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leon","family":"Weber","sequence":"additional","affiliation":[{"name":"Computer Science Department, Humboldt-Universit\u00e4t zu Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mariana","family":"Neves","sequence":"additional","affiliation":[{"name":"Enterprise Platform and Integration Concepts, Hasso-Plattner-Institute, Potsdam, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David Luis","family":"Wiegandt","sequence":"additional","affiliation":[{"name":"Computer Science Department, Humboldt-Universit\u00e4t zu Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ulf","family":"Leser","sequence":"additional","affiliation":[{"name":"Computer Science Department, Humboldt-Universit\u00e4t zu Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2017,7,12]]},"reference":[{"key":"2023051506492196700_btx228-B1","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1038\/nbt1203","article-title":"Gene prioritization through genomic data fusion","volume":"24","author":"Aerts","year":"2006","journal-title":"Nat. Biotechnol"},{"key":"2023051506492196700_btx228-B2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pone.0107477","article-title":"Annotated chemical patent corpus: a gold standard for text mining","volume":"9","author":"Akhondi","year":"2014","journal-title":"PloS One"},{"key":"2023051506492196700_btx228-B3","doi-asserted-by":"crossref","DOI":"10.12688\/f1000research.4591.2","article-title":"Detecting miRNA mentions and relations in biomedical literature","volume":"3","author":"Bagewadi","year":"2014","journal-title":"F1000Research"},{"key":"2023051506492196700_btx228-B4","doi-asserted-by":"crossref","DOI":"10.1186\/1758-2946-7-S1-S6","article-title":"Optimising chemical named entity recognition with pre-processing analytics, knowledge-rich features and heuristics","volume":"7","author":"Batista-Navarro","year":"2015","journal-title":"J. Cheminform"},{"key":"2023051506492196700_btx228-B5","first-page":"175","volume-title":"Theory and Applications for Advanced Text Mining, Chapter Biomedical Named Entity Recognition: A Survey of Machine-Learning Tools","author":"Campos","year":"2012"},{"key":"2023051506492196700_btx228-B6","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-14-54","article-title":"Gimli: open source and high-performance biomedical name recognition","volume":"14","author":"Campos","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023051506492196700_btx228-B7","author":"Chalapathy","year":"2016"},{"key":"2023051506492196700_btx228-B8","author":"Chalapathy","year":"2016"},{"key":"2023051506492196700_btx228-B9","author":"Culotta","year":"2004"},{"key":"2023051506492196700_btx228-B10","author":"Dai","year":"2015"},{"key":"2023051506492196700_btx228-B11","author":"Ding","year":"2002"},{"key":"2023051506492196700_btx228-B12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Do\u011fan","year":"2014","journal-title":"J. Biomed. Inform"},{"key":"2023051506492196700_btx228-B13","doi-asserted-by":"crossref","DOI":"10.1186\/1758-2946-6-17","article-title":"Chemical named entities recognition: a review on approaches and applications","volume":"6","author":"Eltyeb","year":"2014","journal-title":"J. Cheminform"},{"key":"2023051506492196700_btx228-B15","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-9-84","article-title":"OSIRISv1. 2: a named entity recognition system for sequence variants of genes in biomedical literature","volume":"9","author":"Furlong","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2023051506492196700_btx228-B16","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-11-85","article-title":"LINNAEUS: a species name identification system for biomedical literature","volume":"11","author":"Gerner","year":"2010","journal-title":"BMC Bioinformatics"},{"key":"2023051506492196700_btx228-B17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1753-6561-9-S5-A4","article-title":"Linked annotations: a middle ground for manual curation of biomedical databases and text corpora","volume":"9","author":"Goldberg","year":"2015","journal-title":"BMC Proc"},{"key":"2023051506492196700_btx228-B18","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Netw"},{"key":"2023051506492196700_btx228-B19","author":"Gurulingappa","year":"2010"},{"key":"2023051506492196700_btx228-B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13321-016-0172-0","article-title":"Recognizing chemicals in patents - a comparative analysis","volume":"8","author":"Habibi","year":"2016","journal-title":"J. Cheminform"},{"key":"2023051506492196700_btx228-B21","author":"Hahn","year":"2010"},{"key":"2023051506492196700_btx228-B14","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-21606-5","volume-title":"The Elements of Statistical Learning","author":"Hastie","year":"2001"},{"key":"2023051506492196700_btx228-B22","doi-asserted-by":"crossref","first-page":"2983","DOI":"10.1093\/bioinformatics\/btp535","article-title":"A dictionary to identify small molecules and drugs in free text","volume":"25","author":"Hettne","year":"2009","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B23","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"2023051506492196700_btx228-B24","author":"Jozefowicz","year":"2016"},{"key":"2023051506492196700_btx228-B25","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1093\/bioinformatics\/btv570","article-title":"Cell line name recognition in support of the identification of synthetic lethality in cancer from text","volume":"32","author":"Kaewphan","year":"2016","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B26","author":"Kim","year":"2004"},{"key":"2023051506492196700_btx228-B27","author":"Krallinger","year":"2013"},{"key":"2023051506492196700_btx228-B28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1758-2946-7-S1-S1","article-title":"The CHEMDNER corpus of chemicals and drugs and its annotation principles","volume":"7","author":"Krallinger","year":"2015","journal-title":"J. Cheminform"},{"key":"2023051506492196700_btx228-B29","author":"Krallinger","year":"2015"},{"key":"2023051506492196700_btx228-B30","author":"Kulick","year":"2004"},{"key":"2023051506492196700_btx228-B31","author":"Lafferty","year":"2001"},{"key":"2023051506492196700_btx228-B32","author":"Lample","year":"2016"},{"key":"2023051506492196700_btx228-B33","first-page":"652","author":"Leaman","year":"2008"},{"key":"2023051506492196700_btx228-B34","doi-asserted-by":"crossref","first-page":"2839","DOI":"10.1093\/bioinformatics\/btw343","article-title":"TaggerOne: Joint named entity recognition and normalization with Semi-Markov models","author":"Leaman","year":"2016","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B35","author":"Leaman","year":"2009"},{"key":"2023051506492196700_btx228-B36","doi-asserted-by":"crossref","first-page":"2909","DOI":"10.1093\/bioinformatics\/btt474","article-title":"DNorm: disease name normalization with pairwise learning to rank","volume":"29","author":"Leaman","year":"2013","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B37","doi-asserted-by":"crossref","DOI":"10.1186\/1758-2946-7-S1-S3","article-title":"tmChem: a high performance approach for chemical named entity recognition and normalization","volume":"7","author":"Leaman","year":"2015","journal-title":"J. Cheminform"},{"key":"2023051506492196700_btx228-B38","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1093\/bib\/6.4.357","article-title":"What makes a gene name? Named entity recognition in the biomedical literature","volume":"6","author":"Leser","year":"2005","journal-title":"Brief. Bioinform"},{"key":"2023051506492196700_btx228-B39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.1004391","article-title":"miRTex: A text mining system for miRNA-gene relation extraction","volume":"11","author":"Li","year":"2015","journal-title":"PLoS Comput. Biol"},{"key":"2023051506492196700_btx228-B40","article-title":"BioCreative V CDR task corpus: a resource for chemical disease relation extraction","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"key":"2023051506492196700_btx228-B41","doi-asserted-by":"crossref","first-page":"790","DOI":"10.3390\/info6040790","article-title":"Drug name recognition: approaches and resources","volume":"6","author":"Liu","year":"2015","journal-title":"Information"},{"key":"2023051506492196700_btx228-B42","doi-asserted-by":"crossref","first-page":"848","DOI":"10.3390\/info6040848","article-title":"Effects of semantic features on machine learning-based drug name recognition systems: word embeddings vs. manually constructed dictionaries","volume":"6","author":"Liu","year":"2015","journal-title":"Information"},{"key":"2023051506492196700_btx228-B43","author":"Mackin","year":"1978"},{"key":"2023051506492196700_btx228-B44","author":"Narayanaswamy","year":"2003"},{"key":"2023051506492196700_btx228-B45","author":"Neves","year":"2012"},{"key":"2023051506492196700_btx228-B46","author":"Okazaki","year":"2007"},{"key":"2023051506492196700_btx228-B47","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0065390","article-title":"The SPECIES and ORGANISMS resources for fast and accurate identification of taxonomic names in text","volume":"8","author":"Pafilis","year":"2013","journal-title":"PLoS One"},{"key":"2023051506492196700_btx228-B48","author":"Pascanu","year":"2014"},{"key":"2023051506492196700_btx228-B49","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-8-50","article-title":"BioInfer: a corpus for information extraction in the biomedical domain","volume":"8","author":"Pyysalo","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023051506492196700_btx228-B50","author":"Pyysalo","year":"2013"},{"key":"2023051506492196700_btx228-B51","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1109\/5.18626","article-title":"A tutorial on hidden markov models and selected applications in speech recognition","volume":"77","author":"Rabiner","year":"1989","journal-title":"Proceedings of the IEEE"},{"key":"2023051506492196700_btx228-B52","doi-asserted-by":"crossref","first-page":"1633","DOI":"10.1093\/bioinformatics\/bts183","article-title":"ChemSpot: a hybrid system for chemical named entity recognition","volume":"28","author":"Rockt\u00e4schel","year":"2012","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B53","author":"Segura-Bedmar","year":"2015"},{"key":"2023051506492196700_btx228-B54","doi-asserted-by":"crossref","first-page":"3191","DOI":"10.1093\/bioinformatics\/bti475","article-title":"ABNER: an open source tool for automatically tagging genes, proteins and other entity names in text","volume":"21","author":"Settles","year":"2005","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B55","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/gb-2008-9-s2-s2","article-title":"Overview of BioCreative II gene mention recognition","volume":"9","author":"Smith","year":"2008","journal-title":"Genome Biol"},{"key":"2023051506492196700_btx228-B56","author":"Stenetorp","year":"2012"},{"key":"2023051506492196700_btx228-B57","first-page":"1","article-title":"Evaluating word representation features in biomedical named entity recognition tasks","volume":"2014","author":"Tang","year":"2014","journal-title":"BioMed Res. Int"},{"key":"2023051506492196700_btx228-B58","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/0165-0114(79)90023-X","article-title":"On the suitability of minimum and product operators for the intersection of fuzzy sets","volume":"2","author":"Thole","year":"1979","journal-title":"Fuzzy Sets Syst"},{"key":"2023051506492196700_btx228-B59","author":"Tjong Kim Sang","year":"2003"},{"key":"2023051506492196700_btx228-B60","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/va challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J. Am. Med. Inform. Assoc"},{"key":"2023051506492196700_btx228-B61","doi-asserted-by":"crossref","DOI":"10.1093\/database\/bat019","article-title":"Annotating the biomedical literature for the human variome","volume":"2013","author":"Verspoor","year":"2013","journal-title":"Database"},{"key":"2023051506492196700_btx228-B62","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1093\/bioinformatics\/btq002","article-title":"Disambiguating the species of biomedical named entities using natural language parsers","volume":"26","author":"Wang","year":"2010","journal-title":"Bioinformatics"},{"key":"2023051506492196700_btx228-B63","doi-asserted-by":"crossref","first-page":"1080","DOI":"10.1038\/nbt.2758","article-title":"Rational drug repositioning by medical genetics","volume":"31","author":"Wang","year":"2013","journal-title":"Nat. Biotechnol"},{"key":"2023051506492196700_btx228-B64","author":"Wei","year":"2015"},{"key":"2023051506492196700_btx228-B65","doi-asserted-by":"crossref","DOI":"10.1038\/ncomms5212","article-title":"Human symptoms-disease network","volume":"5","author":"Zhou","year":"2014","journal-title":"Nat. Commun"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/33\/14\/i37\/50314882\/bioinformatics_33_14_i37.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/33\/14\/i37\/50314882\/bioinformatics_33_14_i37.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T02:50:20Z","timestamp":1684119020000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/33\/14\/i37\/3953940"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,12]]},"references-count":65,"journal-issue":{"issue":"14","published-print":{"date-parts":[[2017,7,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btx228","relation":{"has-review":[{"id-type":"doi","id":"10.3410\/f.730927099.793537967","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,7,15]]},"published":{"date-parts":[[2017,7,12]]}}}