{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T15:06:35Z","timestamp":1772809595514,"version":"3.50.1"},"reference-count":16,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2021,10,12]],"date-time":"2021-10-12T00:00:00Z","timestamp":1633996800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"New Energy and Industrial Technology Development Organization","award":["JPNP20006"],"award-info":[{"award-number":["JPNP20006"]}]},{"name":"Alan Turing Institute and BBSRC","award":["BB\/P025684\/1"],"award-info":[{"award-number":["BB\/P025684\/1"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,1,12]]},"abstract":"Abstract<\/jats:title>\n \n Summary<\/jats:title>\n Large-scale pre-trained language models (PLMs) have advanced state-of-the-art (SOTA) performance on various biomedical text mining tasks. The power of such PLMs can be combined with the advantages of deep generative models. These are examples of these combinations. However, they are trained only on general domain text, and biomedical models are still missing. In this work, we describe BioVAE, the first large-scale pre-trained latent variable language model for the biomedical domain, which uses the OPTIMUS framework to train on large volumes of biomedical text. The model shows SOTA performance on several biomedical text mining tasks when compared to existing publicly available biomedical PLMs. In addition, our model can generate more accurate biomedical sentences than the original OPTIMUS output.<\/jats:p>\n <\/jats:sec>\n \n Availability and implementation<\/jats:title>\n Our source code and pre-trained models are freely available: https:\/\/github.com\/aistairc\/BioVAE.<\/jats:p>\n <\/jats:sec>\n \n Supplementary information<\/jats:title>\n Supplementary data are available at Bioinformatics online.<\/jats:p>\n <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btab702","type":"journal-article","created":{"date-parts":[[2021,10,8]],"date-time":"2021-10-08T14:31:45Z","timestamp":1633703505000},"page":"872-874","source":"Crossref","is-referenced-by-count":8,"title":["BioVAE: a pre-trained latent variable language model for biomedical text mining"],"prefix":"10.1093","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2472-6370","authenticated-orcid":false,"given":"Hai-Long","family":"Trieu","sequence":"first","affiliation":[{"name":"Artificial Intelligence Research Center (AIRC), National Institute of Advanced Industrial Science and Technology (AIST) , Tokyo 135-0064, Japan"},{"name":"National Centre for Text Mining, Computer Science, University of Manchester 131 Princess Street, M7 1DN , UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Makoto","family":"Miwa","sequence":"additional","affiliation":[{"name":"Artificial Intelligence Research Center (AIRC), National Institute of Advanced Industrial Science and Technology (AIST) , Tokyo 135-0064, Japan"},{"name":"Department of Advanced Science and Technology, Toyota Technological Institute, Nagoya 468-8511, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sophia","family":"Ananiadou","sequence":"additional","affiliation":[{"name":"National Centre for Text Mining, Computer Science, University of Manchester 131 Princess Street, M7 1DN , UK"},{"name":"Alan Turing Institute , 96 Euston Road, London, NW1 2DB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2021,10,12]]},"reference":[{"key":"2023020108475719200_btab702-B1","first-page":"3606","author":"Beltagy","year":"2019"},{"key":"2023020108475719200_btab702-B2","first-page":"10","author":"Bowman","year":"2016"},{"key":"2023020108475719200_btab702-B3","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2023020108475719200_btab702-B4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Do\u011fan","year":"2014","journal-title":"Biomed. Inf"},{"key":"2023020108475719200_btab702-B5","author":"Gu","year":"2020"},{"key":"2023020108475719200_btab702-B6","doi-asserted-by":"crossref","first-page":"73729","DOI":"10.1109\/ACCESS.2019.2920708","article-title":"A neural named entity recognition and multi-type normalization tool for biomedical text mining","volume":"7","author":"Kim","year":"2019","journal-title":"IEEE Access"},{"key":"2023020108475719200_btab702-B7","first-page":"70","author":"Kim","year":"2004"},{"key":"2023020108475719200_btab702-B8","author":"Kingma","year":"2014"},{"key":"2023020108475719200_btab702-B9","doi-asserted-by":"crossref","first-page":"bav123","DOI":"10.1093\/database\/bav123","article-title":"Chemprot-3.0: a global chemical biology diseases mapping","volume":"2016","author":"Kringelum","year":"2016","journal-title":"Database"},{"key":"2023020108475719200_btab702-B10","first-page":"36:1234-1240","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2023020108475719200_btab702-B11","first-page":"4678","author":"Li","year":"2020"},{"key":"2023020108475719200_btab702-B12","doi-asserted-by":"crossref","first-page":"baw068","DOI":"10.1093\/database\/baw068","article-title":"Biocreative V CDR task corpus: a resource for chemical disease relation extraction","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"key":"2023020108475719200_btab702-B13","first-page":"65","article-title":"A Bert-based universal model for both within-and cross-sentence clinical temporal relation extraction","author":"Lin","year":"2019"},{"key":"2023020108475719200_btab702-B14","first-page":"553","article-title":"Results of the seventh edition of the bioasq challenge","author":"Nentidis","year":"2019"},{"key":"2023020108475719200_btab702-B15","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2023020108475719200_btab702-B16","article-title":"Large batch optimization for deep learning: training Bert in 76 minutes","author":"You","year":"2020"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btab702\/41135090\/btab702.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/3\/872\/49007494\/btab702.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/3\/872\/49007494\/btab702.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T20:04:43Z","timestamp":1675281883000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/38\/3\/872\/6390793"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,10,12]]},"references-count":16,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,1,12]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btab702","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,2,1]]},"published":{"date-parts":[[2021,10,12]]}}}