{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T16:11:41Z","timestamp":1774455101940,"version":"3.50.1"},"reference-count":68,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2020,6,26]],"date-time":"2020-06-26T00:00:00Z","timestamp":1593129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61702121"],"award-info":[{"award-number":["61702121"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61772378"],"award-info":[{"award-number":["61772378"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002338","name":"Ministry of Education of China","doi-asserted-by":"publisher","award":["18JZD015"],"award-info":[{"award-number":["18JZD015"]}],"id":[{"id":"10.13039\/501100002338","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Key Project of State Language Commission of China","award":["ZDI135-112"],"award-info":[{"award-number":["ZDI135-112"]}]},{"name":"Guangdong Basic and Applied Basic Research Foundation","award":["2020A151501705"],"award-info":[{"award-number":["2020A151501705"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,5,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Biomedical information extraction (BioIE) is an important task. The aim is to analyze biomedical texts and extract structured information such as named entities and semantic relations between them. In recent years, pre-trained language models have largely improved the performance of BioIE. However, they neglect to incorporate external structural knowledge, which can provide rich factual information to support the underlying understanding and reasoning for biomedical information extraction. In this paper, we first evaluate current extraction methods, including vanilla neural networks, general language models and pre-trained contextualized language models on biomedical information extraction tasks, including named entity recognition, relation extraction and event extraction. We then propose to enrich a contextualized language model by integrating a large scale of biomedical knowledge graphs (namely, BioKGLM). In order to effectively encode knowledge, we explore a three-stage training procedure and introduce different fusion strategies to facilitate knowledge injection. Experimental results on multiple tasks show that BioKGLM consistently outperforms state-of-the-art extraction models. A further analysis proves that BioKGLM can capture the underlying relations between biomedical knowledge concepts, which are crucial for BioIE.<\/jats:p>","DOI":"10.1093\/bib\/bbaa110","type":"journal-article","created":{"date-parts":[[2020,5,7]],"date-time":"2020-05-07T19:09:02Z","timestamp":1588878542000},"source":"Crossref","is-referenced-by-count":97,"title":["Enriching contextualized language model from knowledge graph for biomedical information extraction"],"prefix":"10.1093","volume":"22","author":[{"given":"Hao","family":"Fei","sequence":"first","affiliation":[]},{"given":"Yafeng","family":"Ren","sequence":"additional","affiliation":[]},{"given":"Yue","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Donghong","family":"Ji","sequence":"additional","affiliation":[]},{"given":"Xiaohui","family":"Liang","sequence":"additional","affiliation":[]}],"member":"286","published-online":{"date-parts":[[2020,6,26]]},"reference":[{"key":"2021052110025637400_ref1","first-page":"224","article-title":"Identifying protein-protein interaction using tree lstm and structured attention","volume-title":"Proceedings of 2019 IEEE International Conference on Semantic Computing","author":"Ahmed","year":"2019"},{"key":"2021052110025637400_ref2","article-title":"Publicly available clinical BERT embeddings","author":"Alsentzer","year":"2019","journal-title":"CoRR"},{"issue":"7","key":"2021052110025637400_ref3","doi-asserted-by":"crossref","first-page":"e0200699","DOI":"10.1371\/journal.pone.0200699","article-title":"Automatic extraction of gene-disease associations from literature using joint ensemble learning","volume":"13","author":"Bhasuran","year":"2018","journal-title":"PloS One"},{"key":"2021052110025637400_ref4","first-page":"183","article-title":"Generalizing biomedical event extraction","author":"Bj\u00f6rne","year":"2011","journal-title":"Proceedings of the BioNLP Shared Task 2011 Workshop"},{"key":"2021052110025637400_ref5","doi-asserted-by":"crossref","first-page":"98","DOI":"10.18653\/v1\/W18-2311","article-title":"Biomedical event extraction using convolutional neural networks and dependency parsing","author":"Bj\u00f6rne","year":"2018","journal-title":"Proceedings of the BioNLP 2018 Workshop"},{"key":"2021052110025637400_ref6","first-page":"2787","article-title":"Translating embeddings for modeling multi-relational data","author":"Bordes","year":"2013","journal-title":"Proceedings of Advances in Neural Information Processing Systems"},{"issue":"1","key":"2021052110025637400_ref7","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s12859-015-0472-9","article-title":"Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research","volume":"16","author":"Bravo","year":"2015","journal-title":"BMC Bioinformatics"},{"key":"2021052110025637400_ref8","doi-asserted-by":"crossref","first-page":"227","DOI":"10.18653\/v1\/D18-1021","article-title":"Joint representation learning of cross-lingual words and entities via attentive distant supervision","author":"Cao","year":"2018","journal-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"},{"key":"2021052110025637400_ref9","first-page":"1623","article-title":"Bridge text and knowledge by learning multi-prototype entity mention embedding","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","author":"Cao","year":"2017"},{"key":"2021052110025637400_ref10","first-page":"2406","article-title":"Neural natural language inference models enhanced with external knowledge","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Chen","year":"2018"},{"issue":"1","key":"2021052110025637400_ref11","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1186\/s12911-018-0655-1","article-title":"Correction to: evaluating semantic relations in neural word embeddings with biomedical and general domain knowledge bases","volume":"18","author":"Chen","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021052110025637400_ref12","doi-asserted-by":"crossref","DOI":"10.3115\/v1\/D14-1179","article-title":"Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation","author":"Cho","year":"2014","journal-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing"},{"key":"2021052110025637400_ref13","first-page":"2126","article-title":"What you can cram into a single $\\$\\&!\\#{\\ast }$ vector: probing sentence embeddings for linguistic properties","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Conneau","year":"2018"},{"issue":"D1","key":"2021052110025637400_ref14","doi-asserted-by":"crossref","first-page":"D204","DOI":"10.1093\/nar\/gku989","article-title":"Uniprot: a hub for protein information","volume":"43","author":"UniProt Consortium","year":"2014","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"2021052110025637400_ref15","doi-asserted-by":"crossref","first-page":"D472","DOI":"10.1093\/nar\/gkt1102","article-title":"The reactome pathway knowledgebase","volume":"42","author":"Croft","year":"2013","journal-title":"Nucleic Acids Res"},{"issue":"20","key":"2021052110025637400_ref16","doi-asserted-by":"crossref","first-page":"3539","DOI":"10.1093\/bioinformatics\/bty356","article-title":"D3ner: biomedical named entity recognition using crf-bilstm improved with fine-tuned embeddings of various linguistic information","volume":"34","author":"Dang","year":"2018","journal-title":"Bioinformatics"},{"key":"2021052110025637400_ref17","doi-asserted-by":"crossref","first-page":"12","DOI":"10.18653\/v1\/W16-3002","article-title":"Overview of the bacteria biotope task at bionlp shared task 2016","author":"Del\u00e9ger","year":"2016","journal-title":"Proceedings of the 4th BioNLP Shared Task Workshop"},{"key":"2021052110025637400_ref18","first-page":"4171","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics","author":"Devlin","year":"2019"},{"key":"2021052110025637400_ref19","doi-asserted-by":"crossref","first-page":"1795","DOI":"10.1109\/BIBM.2017.8217933","article-title":"Drug\u2013drug interaction relation extraction with deep convolutional neural networks","volume-title":"Proceedings of 2017 IEEE International Conference on Bioinformatics and Biomedicine","author":"Dewi","year":"2017"},{"key":"2021052110025637400_ref20","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1016\/j.ins.2019.09.075","article-title":"A tree-based neural network model for biomedical event trigger detection","volume":"512","author":"Fei","year":"2020","journal-title":"Inform Sci"},{"issue":"1","key":"2021052110025637400_ref21","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1186\/s12859-017-1609-9","article-title":"A neural joint model for entity and relation extraction from biomedical text","volume":"18","author":"Fei","year":"2017","journal-title":"BMC Bioinformatics"},{"issue":"23","key":"2021052110025637400_ref22","doi-asserted-by":"crossref","first-page":"4087","DOI":"10.1093\/bioinformatics\/bty449","article-title":"Transfer learning for biomedical named entity recognition with neural networks","volume":"34","author":"Giorgi","year":"2018","journal-title":"Bioinformatics"},{"key":"2021052110025637400_ref23","article-title":"Assessing bert\u2019s syntactic abilities","author":"Goldberg","year":"2019","journal-title":"CoRR"},{"issue":"D1","key":"2021052110025637400_ref24","doi-asserted-by":"crossref","first-page":"D1079","DOI":"10.1093\/nar\/gku1071","article-title":"Genenames.Org: the hgnc resources in 2015","volume":"43","author":"Gray","year":"2014","journal-title":"Nucleic Acids Res"},{"issue":"5","key":"2021052110025637400_ref25","doi-asserted-by":"crossref","first-page":"885","DOI":"10.1016\/j.jbi.2012.04.008","article-title":"Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports","volume":"45","author":"Gurulingappa","year":"2012","journal-title":"J Biomed Inform"},{"key":"2021052110025637400_ref26","article-title":"Joint representation learning of text and knowledge for knowledge graph completion","author":"Xu","year":"2016","journal-title":"CoRR"},{"key":"2021052110025637400_ref27","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1356","article-title":"What does bert learn about the structure of language?","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Jawahar","year":"2019"},{"key":"2021052110025637400_ref28","first-page":"687","article-title":"Knowledge graph embedding via dynamic mapping matrix","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","author":"Ji","year":"2015"},{"key":"2021052110025637400_ref29","article-title":"Probing biomedical embeddings from language models","author":"Jin","year":"2019","journal-title":"CoRR"},{"issue":"23","key":"2021052110025637400_ref30","doi-asserted-by":"crossref","first-page":"3158","DOI":"10.1093\/bioinformatics\/bts591","article-title":"Semmeddb: a pubmed-scale repository of biomedical semantic predications","volume":"28","author":"Kilicoglu","year":"2012","journal-title":"Bioinformatics"},{"key":"2021052110025637400_ref31","first-page":"70","article-title":"Introduction to the bio-entity recognition task at jnlpba","author":"Kim","year":"2004","journal-title":"Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications"},{"key":"2021052110025637400_ref32","doi-asserted-by":"crossref","first-page":"298","DOI":"10.18653\/v1\/W19-5032","article-title":"Embedding biomedical ontologies by jointly encoding network structure and textual node descriptors","author":"Kotitsas","year":"2019","journal-title":"Proceedings of the 18th BioNLP Workshop and Shared Task"},{"key":"2021052110025637400_ref33","article-title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","author":"Lee","year":"2019","journal-title":"CoRR"},{"key":"2021052110025637400_ref34","doi-asserted-by":"crossref","first-page":"42","DOI":"10.18653\/v1\/W16-3005","article-title":"Verse: event and relation extraction in the bionlp 2016 shared task","author":"Lever","year":"2016","journal-title":"Proceedings of the 4th BioNLP Shared Task Workshop"},{"key":"2021052110025637400_ref35","first-page":"1421","article-title":"Biomedical event extraction based on knowledge-driven tree-lstm","author":"Li","year":"2019","journal-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics"},{"key":"2021052110025637400_ref36","first-page":"4171","article-title":"Biocreative v cdr task corpus: a resource for chemical disease relation extraction","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"issue":"9","key":"2021052110025637400_ref37","first-page":"177","article-title":"Biomedical event extraction based on gru integrating attention mechanism","volume":"19","author":"Li","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2021052110025637400_ref38","first-page":"2181","article-title":"Learning entity and relation embeddings for knowledge graph completion","author":"Lin","year":"2015","journal-title":"Proceedings of the Twenty-ninth AAAI Conference on Artificial Intelligence"},{"key":"2021052110025637400_ref39","first-page":"968","article-title":"Integrating extra knowledge into word embedding models for biomedical NLP tasks","volume-title":"Proceedings of the 2017 International Joint Conference on Neural Networks","author":"Ling","year":"2017"},{"issue":"Nov","key":"2021052110025637400_ref40","first-page":"2579","article-title":"Visualizing data using t-sne","volume":"9","author":"van der Maaten","year":"2008","journal-title":"J Mach Learn Res"},{"key":"2021052110025637400_ref41","first-page":"1468","article-title":"Mem2Seq: effectively incorporating knowledge bases into end-to-end task-oriented dialog systems","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Madotto","year":"2018"},{"key":"2021052110025637400_ref42","first-page":"299","volume-title":"Biograkn: a knowledge graph-based semantic database for biomedical sciences","author":"Messina","year":"2017"},{"key":"2021052110025637400_ref43","first-page":"821","article-title":"Knowledgeable reader: enhancing cloze-style reading comprehension with external commonsense knowledge","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Mihaylov","year":"2018"},{"key":"2021052110025637400_ref44","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov","year":"2013"},{"key":"2021052110025637400_ref45","first-page":"240","article-title":"Link prediction using multi part embeddings","volume-title":"Proceedings of the 16th International Conference on Semantic Web","author":"Mohamed","year":"2019"},{"key":"2021052110025637400_ref46","first-page":"21","article-title":"An extended dependency graph for relation extraction in biomedical texts","author":"Peng","year":"2015","journal-title":"Proceedings of BioNLP Shared Task Workshop"},{"key":"2021052110025637400_ref47","first-page":"1532","article-title":"Glove: global vectors for word representation","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","author":"Pennington","year":"2014"},{"key":"2021052110025637400_ref48","first-page":"2227","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics","author":"Peters","year":"2018"},{"issue":"1","key":"2021052110025637400_ref49","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1186\/1471-2105-8-50","article-title":"Bioinfer: a corpus for information extraction in the biomedical domain","volume":"8","author":"Pyysalo","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2021052110025637400_ref50","article-title":"Improving language understanding by generative pre-training","author":"Radford","year":"2018"},{"key":"2021052110025637400_ref51","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1109\/BIBM47256.2019.8983248","article-title":"Drug\u2013drug interaction extraction using a span-based neural network model","author":"Ren","year":"2019","journal-title":"Proceedings of 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)"},{"key":"2021052110025637400_ref52","first-page":"1599","article-title":"Cross-lingual alignment of contextual word embeddings with applications to zero-shot dependency parsing","author":"Schuster","year":"2019","journal-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics"},{"issue":"65","key":"2021052110025637400_ref53","first-page":"341","article-title":"Semeval-2013 task 9: extraction of drug\u2013drug interactions from biomedical texts","volume":"3206","author":"Segura-Bedmar","year":"2013","journal-title":"Assoc Comput Linguist"},{"key":"2021052110025637400_ref54","doi-asserted-by":"crossref","first-page":"1662","DOI":"10.1109\/BIBM.2018.8621405","article-title":"Deep convolution neural networks for drug\u2013drug interaction extraction","author":"Sun","year":"2018","journal-title":"Proceedings of 2018 IEEE International Conference on Bioinformatics and Biomedicine"},{"key":"2021052110025637400_ref55","article-title":"ERNIE: enhanced representation through knowledge integration","author":"Yu","year":"2019","journal-title":"CoRR"},{"key":"2021052110025637400_ref56","article-title":"What do you learn from context? probing for sentence structure in contextualized word representations","volume-title":"Proceedings of the 7th International Conference on Learning Representations","author":"Tenney","year":"2019"},{"key":"2021052110025637400_ref57","first-page":"2071","article-title":"Complex embeddings for simple link prediction","volume-title":"Proceedings of the 33nd International Conference on Machine Learning","author":"Trouillon","year":"2016"},{"key":"2021052110025637400_ref58","first-page":"5998","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Proceedings of Advances in Neural Information Processing Systems"},{"key":"2021052110025637400_ref59","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/W19-4808","article-title":"Analyzing the structure of attention in a transformer language model","author":"Vig","year":"2019"},{"issue":"10","key":"2021052110025637400_ref60","doi-asserted-by":"crossref","first-page":"1745","DOI":"10.1093\/bioinformatics\/bty869","article-title":"Cross-type biomedical named entity recognition with deep multi-task learning","volume":"35","author":"Wang","year":"2018","journal-title":"Bioinformatics"},{"key":"2021052110025637400_ref61","doi-asserted-by":"crossref","first-page":"1591","DOI":"10.3115\/v1\/D14-1167","article-title":"Knowledge graph and text jointly embedding","author":"Wang","year":"2014","journal-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing"},{"key":"2021052110025637400_ref62","first-page":"1112","article-title":"Knowledge graph embedding by translating on hyperplanes","author":"Wang","year":"2014","journal-title":"Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence"},{"key":"2021052110025637400_ref63","article-title":"Embedding entities and relations for learning and inference in knowledge bases","author":"Yang","year":"2015","journal-title":"Proceedings of the 3rd International Conference on Learning Representations"},{"key":"2021052110025637400_ref64","article-title":"Xlnet: generalized autoregressive pretraining for language understanding","author":"Yang","year":"2019","journal-title":"CoRR"},{"issue":"10","key":"2021052110025637400_ref65","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1186\/s12859-019-2813-6","article-title":"Collabonet: collaboration of deep neural networks for biomedical named entity recognition","volume":"20","author":"Yoon","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2021052110025637400_ref66","first-page":"656","article-title":"Adaptive knowledge sharing in multi-task learning: improving low-resource neural machine translation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Zaremoodi","year":"2018"},{"issue":"6","key":"2021052110025637400_ref67","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.1016\/j.jbi.2011.08.011","article-title":"Neighborhood hash graph kernel for protein\u2013protein interaction extraction","volume":"44","author":"Zhang","year":"2011","journal-title":"J Biomed Inform"},{"key":"2021052110025637400_ref68","doi-asserted-by":"crossref","first-page":"1441","DOI":"10.18653\/v1\/P19-1139","article-title":"ERNIE: enhanced language representation with informative entities","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Zhang","year":"2019"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/3\/bbaa110\/37963251\/bbaa110.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/3\/bbaa110\/37963251\/bbaa110.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,21]],"date-time":"2021-05-21T10:04:17Z","timestamp":1621591457000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaa110\/5854405"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,26]]},"references-count":68,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,5,20]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaa110","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,5]]},"published":{"date-parts":[[2020,6,26]]},"article-number":"bbaa110"}}