{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T22:26:34Z","timestamp":1785018394282,"version":"3.55.0"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2019,9,10]],"date-time":"2019-09-10T00:00:00Z","timestamp":1568073600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research Foundation of Korea(NRF) funded by the Korea government","award":["NRF-2017R1A2A1A17069645"],"award-info":[{"award-number":["NRF-2017R1A2A1A17069645"]}]},{"name":"National Research Foundation of Korea(NRF) funded by the Korea government","award":["NRF-2017M3C4A7065887"],"award-info":[{"award-number":["NRF-2017M3C4A7065887"]}]},{"name":"National Research Foundation of Korea(NRF) funded by the Korea government","award":["NRF-2014M3C9A3063541"],"award-info":[{"award-number":["NRF-2014M3C9A3063541"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,2,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>We make the pre-trained weights of BioBERT freely available at https:\/\/github.com\/naver\/biobert-pretrained, and the source code for fine-tuning BioBERT available at https:\/\/github.com\/dmis-lab\/biobert.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btz682","type":"journal-article","created":{"date-parts":[[2019,9,5]],"date-time":"2019-09-05T15:27:43Z","timestamp":1567697263000},"page":"1234-1240","source":"Crossref","is-referenced-by-count":5443,"title":["BioBERT: a pre-trained biomedical language representation model for biomedical text mining"],"prefix":"10.1093","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4972-239X","authenticated-orcid":false,"given":"Jinhyuk","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University , Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6435-548X","authenticated-orcid":false,"given":"Wonjin","family":"Yoon","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University , Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0240-6210","authenticated-orcid":false,"given":"Sungdong","family":"Kim","sequence":"additional","affiliation":[{"name":"Clova AI Research, Naver Corp , Seong-Nam 13561, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8224-8354","authenticated-orcid":false,"given":"Donghyeon","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University , Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0240-6210","authenticated-orcid":false,"given":"Sunkyu","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University , Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7633-1074","authenticated-orcid":false,"given":"Chan Ho","family":"So","sequence":"additional","affiliation":[{"name":"Interdisciplinary Graduate Program in Bioinformatics, Korea University , Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6798-9106","authenticated-orcid":false,"given":"Jaewoo","family":"Kang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Korea University , Seoul 02841, Korea"},{"name":"Interdisciplinary Graduate Program in Bioinformatics, Korea University , Seoul 02841, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,9,10]]},"reference":[{"key":"2023013110145762300_btz682-B1","doi-asserted-by":"crossref","first-page":"72","DOI":"10.18653\/v1\/W19-1909","article-title":"Publicly available clinical bert embeddings","author":"Alsentzer","year":"2019","journal-title":"Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, MN, USA"},{"key":"2023013110145762300_btz682-B2","doi-asserted-by":"crossref","first-page":"e0200699","DOI":"10.1371\/journal.pone.0200699","article-title":"Automatic extraction of gene-disease associations from literature using joint ensemble learning","volume":"13","author":"Bhasuran","year":"2018","journal-title":"PLoS One"},{"key":"2023013110145762300_btz682-B3","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s12859-015-0472-9","article-title":"Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research","volume":"16","author":"Bravo","year":"2015","journal-title":"BMC Bioinformatics"},{"key":"2023013110145762300_btz682-B4","first-page":"4171","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019","journal-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA"},{"key":"2023013110145762300_btz682-B5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Do\u011fan","year":"2014","journal-title":"J. Biomed. Inform"},{"key":"2023013110145762300_btz682-B6","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/1471-2105-11-85","article-title":"Linnaeus: a species name identification system for biomedical literature","volume":"11","author":"Gerner","year":"2010","journal-title":"BMC Bioinformatics"},{"key":"2023013110145762300_btz682-B7","doi-asserted-by":"crossref","first-page":"4087","DOI":"10.1093\/bioinformatics\/bty449","article-title":"Transfer learning for biomedical named entity recognition with neural networks","volume":"34","author":"Giorgi","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013110145762300_btz682-B8","doi-asserted-by":"crossref","first-page":"i37","DOI":"10.1093\/bioinformatics\/btx228","article-title":"Deep learning with word embeddings improves biomedical named entity recognition","volume":"33","author":"Habibi","year":"2017","journal-title":"Bioinformatics"},{"key":"2023013110145762300_btz682-B9","author":"Kim","year":"2004"},{"key":"2023013110145762300_btz682-B10","doi-asserted-by":"crossref","DOI":"10.1186\/1758-2946-7-S1-S2","article-title":"The chemdner corpus of chemicals and drugs and its annotation principles","volume":"7","author":"Krallinger","year":"2015","journal-title":"J. Cheminform"},{"key":"2023013110145762300_btz682-B11","first-page":"141","article-title":"Overview of the BioCreative VI chemical-protein interaction track","author":"Krallinger","year":"2017","journal-title":"Proceedings of the BioCreative VI Workshop, Bethesda, MD, USA"},{"key":"2023013110145762300_btz682-B12","article-title":"Biocreative V CDR task corpus: a resource for chemical disease relation extraction","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"key":"2023013110145762300_btz682-B13","doi-asserted-by":"crossref","DOI":"10.1093\/database\/bay060","article-title":"Chemical\u2013gene relation extraction using recursive neural network","volume":"2018","author":"Lim","year":"2018","journal-title":"Database"},{"key":"2023013110145762300_btz682-B14","doi-asserted-by":"crossref","first-page":"65","DOI":"10.18653\/v1\/W19-1908","article-title":"A bert-based universal model for both within-and cross-sentence clinical temporal relation extraction","author":"Lin","year":"2019","journal-title":"Proceedings of the 2nd Clinical Natural Language Processing Workshop, Minneapolis, MN, USA."},{"key":"2023013110145762300_btz682-B15","doi-asserted-by":"crossref","first-page":"2363","DOI":"10.1093\/bioinformatics\/btx172","article-title":"A transition-based joint model for disease named entity recognition and normalization","volume":"33","author":"Lou","year":"2017","journal-title":"Bioinformatics"},{"key":"2023013110145762300_btz682-B16","doi-asserted-by":"crossref","first-page":"1381","DOI":"10.1093\/bioinformatics\/btx761","article-title":"An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition","volume":"34","author":"Luo","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013110145762300_btz682-B17","first-page":"6294","article-title":"Learned in translation: contextualized word vectors","author":"McCann","year":"2017","journal-title":"Advances in Neural Information Processing Systems 30"},{"key":"2023013110145762300_btz682-B18","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","author":"Mikolov","year":"2013","journal-title":"Advances in Neural Information Processing Systems 26"},{"key":"2023013110145762300_btz682-B19","author":"Mohan","year":"2019"},{"key":"2023013110145762300_btz682-B20","doi-asserted-by":"crossref","first-page":"e65390","DOI":"10.1371\/journal.pone.0065390","article-title":"The species and organisms resources for fast and accurate identification of taxonomic names in text","volume":"8","author":"Pafilis","year":"2013","journal-title":"PLoS One"},{"key":"2023013110145762300_btz682-B21","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"Glove: Global vectors for word representation","author":"Pennington","year":"2014","journal-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar"},{"key":"2023013110145762300_btz682-B22","author":"Peters","year":"2018"},{"key":"2023013110145762300_btz682-B23","first-page":"39","article-title":"Distributional semantics resources for biomedical text processing","author":"Pyysalo","year":"2013","journal-title":"Proceedings of the 5th International Symposium on Languages in Biology and Medicine, Tokyo, Japan"},{"key":"2023013110145762300_btz682-B24","author":"Rajpurkar","year":"2016"},{"key":"2023013110145762300_btz682-B25","author":"Sachan","year":"2018"},{"key":"2023013110145762300_btz682-B26","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/gb-2008-9-s2-s2","article-title":"Overview of biocreative ii gene mention recognition","volume":"9","author":"Smith","year":"2008","journal-title":"Genome Biol"},{"key":"2023013110145762300_btz682-B27","first-page":"1487","article-title":"A silver standard corpus of human phenotype-gene relations","author":"Sousa","year":"2019","journal-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN"},{"key":"2023013110145762300_btz682-B28","author":"Sung","year":"2017"},{"key":"2023013110145762300_btz682-B29","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1186\/s12859-015-0564-6","article-title":"An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition","volume":"16","author":"Tsatsaronis","year":"2015","journal-title":"BMC Bioinformatics"},{"key":"2023013110145762300_btz682-B30","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J. Am. Med. Inform. Assoc"},{"key":"2023013110145762300_btz682-B31","doi-asserted-by":"crossref","first-page":"879","DOI":"10.1016\/j.jbi.2012.04.004","article-title":"The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships","volume":"45","author":"Van Mulligen","year":"2012","journal-title":"J. Biomed. Inform"},{"key":"2023013110145762300_btz682-B32","first-page":"5998","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2023013110145762300_btz682-B33","author":"Wang","year":"2018"},{"key":"2023013110145762300_btz682-B34","author":"Wiese","year":"2017"},{"key":"2023013110145762300_btz682-B35","author":"Wu","year":"2016"},{"key":"2023013110145762300_btz682-B36","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.compbiomed.2019.04.002","article-title":"Document-level attention-based BiLSTM-CRF incorporating disease dictionary for disease named entity recognition","volume":"108","author":"Xu","year":"2019","journal-title":"Comput. Biol. Med"},{"key":"2023013110145762300_btz682-B37","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1186\/s12859-019-2813-6","article-title":"Collabonet: collaboration of deep neural networks for biomedical named entity recognition","volume":"20","author":"Yoon","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2023013110145762300_btz682-B38","author":"Zhu","year":"2018"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btz682\/30132027\/btz682.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/4\/1234\/48983216\/bioinformatics_36_4_1234.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/4\/1234\/48983216\/bioinformatics_36_4_1234.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T15:16:03Z","timestamp":1675178163000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/36\/4\/1234\/5566506"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2019,9,10]]},"references-count":38,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,2,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btz682","relation":{"has-review":[{"id-type":"doi","id":"10.3410\/f.736581786.793584048","asserted-by":"object"},{"id-type":"doi","id":"10.3410\/f.736581786.793583871","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,2,15]]},"published":{"date-parts":[[2019,9,10]]}}}