{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,10]],"date-time":"2026-08-10T15:22:07Z","timestamp":1786375327169,"version":"3.56.0"},"reference-count":72,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2020,10,29]],"date-time":"2020-10-29T00:00:00Z","timestamp":1603929600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"Patient-Centered Outcomes Research Institute Award","award":["ME-2018C3-14754"],"award-info":[{"award-number":["ME-2018C3-14754"]}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["1R01CA246418 R01"],"award-info":[{"award-number":["1R01CA246418 R01"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["NIA R21AG062884"],"award-info":[{"award-number":["NIA R21AG062884"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"name":"University of Florida Informatics Institute Junior SEED Program","award":["00129436"],"award-info":[{"award-number":["00129436"]}]},{"name":"Cancer Informatics and eHealth"},{"name":"University of Florida Health Cancer Center"},{"name":"University of Florida Clinical and Translational Science Institute"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,12,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>The goal of this study is to explore transformer-based models (eg, Bidirectional Encoder Representations from Transformers [BERT]) for clinical concept extraction and develop an open-source package with pretrained clinical models to facilitate concept extraction and other downstream natural language processing (NLP) tasks in the medical domain.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We systematically explored 4 widely used transformer-based architectures, including BERT, RoBERTa, ALBERT, and ELECTRA, for extracting various types of clinical concepts using 3 public datasets from the 2010 and 2012 i2b2 challenges and the 2018 n2c2 challenge. We examined general transformer models pretrained using general English corpora as well as clinical transformer models pretrained using a clinical corpus and compared them with a long short-term memory conditional random fields (LSTM-CRFs) mode as a baseline. Furthermore, we integrated the 4 clinical transformer-based models into an open-source package.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results and Conclusion<\/jats:title>\n                  <jats:p>The RoBERTa-MIMIC model achieved state-of-the-art performance on 3 public clinical concept extraction datasets with F1-scores of 0.8994, 0.8053, and 0.8907, respectively. Compared to the baseline LSTM-CRFs model, RoBERTa-MIMIC remarkably improved the F1-score by approximately 4% and 6% on the 2010 and 2012 i2b2 datasets. This study demonstrated the efficiency of transformer-based models for clinical concept extraction. Our methods and systems can be applied to other clinical tasks. The clinical transformer package with 4 pretrained clinical models is publicly available at https:\/\/github.com\/uf-hobi-informatics-lab\/ClinicalTransformerNER. We believe this package will improve current practice on clinical concept extraction and other tasks in the medical domain.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa189","type":"journal-article","created":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T11:09:33Z","timestamp":1595675373000},"page":"1935-1942","source":"Crossref","is-referenced-by-count":125,"title":["Clinical concept extraction using transformers"],"prefix":"10.1093","volume":"27","author":[{"given":"Xi","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"},{"name":"Cancer Informatics and eHealth core, University of Florida Health Cancer Center, Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiang","family":"Bian","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"},{"name":"Cancer Informatics and eHealth core, University of Florida Health Cancer Center, Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9881-1017","authenticated-orcid":false,"given":"William R","family":"Hogan","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yonghui","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA"},{"name":"Cancer Informatics and eHealth core, University of Florida Health Cancer Center, Gainesville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,10,29]]},"reference":[{"key":"2020121009244545000_ocaa189-B1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"2020121009244545000_ocaa189-B2","first-page":"104","article-title":"Systematic analysis of free-text family history in electronic health record","volume":"2017","author":"Wang","year":"2017","journal-title":"AMIA Jt Summ Transl Sci Proc"},{"issue":"11","key":"2020121009244545000_ocaa189-B3","doi-asserted-by":"crossref","first-page":"1075","DOI":"10.1007\/s40264-017-0558-6","article-title":"Natural language processing for EHR-based pharmacovigilance: a structured review","volume":"40","author":"Luo","year":"2017","journal-title":"Drug Saf"},{"key":"2020121009244545000_ocaa189-B4","volume-title":"Capturing Social and Behavioral Domains in Electronic Health Records: Phase 1","year":"2014"},{"key":"2020121009244545000_ocaa189-B5","volume-title":"Capturing Social and Behavioral Domains and Measures in Electronic Health Records: Phase 2","year":"2015"},{"key":"2020121009244545000_ocaa189-B6","author":"Fu"},{"key":"2020121009244545000_ocaa189-B7","first-page":"76","article-title":"Open Source Clinical NLP\u2013more than any single system","volume":"2014","author":"Masanz","year":"2014","journal-title":"AMIA Jt Summ Transl Sci Proc"},{"issue":"5","key":"2020121009244545000_ocaa189-B8","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","article-title":"Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis","volume":"22","author":"Shickel","year":"2018","journal-title":"IEEE J Biomed Health Inform"},{"key":"2020121009244545000_ocaa189-B9","author":"Li","year":"2018"},{"issue":"3","key":"2020121009244545000_ocaa189-B10","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1093\/jamia\/ocz200","article-title":"Deep learning in clinical natural language processing: a methodical review","volume":"27","author":"Wu","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244545000_ocaa189-B11","author":"Wolf","year":"2019"},{"key":"2020121009244545000_ocaa189-B12","author":"Devlin","year":"2018"},{"key":"2020121009244545000_ocaa189-B13","volume-title":"arXiv:190608237 [cs]","author":"Yang"},{"key":"2020121009244545000_ocaa189-B14","author":"Lan","year":"2019"},{"key":"2020121009244545000_ocaa189-B15","author":"Liu","year":"2019"},{"key":"2020121009244545000_ocaa189-B16","author":"Raffel"},{"key":"2020121009244545000_ocaa189-B17","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","author":"Lee","year":"2019","journal-title":"Bioinformatics"},{"issue":"11","key":"2020121009244545000_ocaa189-B18","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1093\/jamia\/ocz096","article-title":"Enhancing clinical concept extraction with contextual embeddings","volume":"26","author":"Si","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244545000_ocaa189-B19","author":"Huang"},{"key":"2020121009244545000_ocaa189-B20","author":"Peng","year":"2019"},{"key":"2020121009244545000_ocaa189-B21","first-page":"72","author":"Alsentzer","year":"2019"},{"key":"2020121009244545000_ocaa189-B22","author":"Clark","year":"2020"},{"issue":"5","key":"2020121009244545000_ocaa189-B23","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020121009244545000_ocaa189-B24","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 Challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020121009244545000_ocaa189-B25","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/jamia\/ocz166","article-title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records","volume":"27","author":"Henry","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244545000_ocaa189-B26","first-page":"347","author":"Friedman"},{"key":"2020121009244545000_ocaa189-B27","first-page":"270","article-title":"A broad-coverage natural language processing system","volume":"2000","author":"Friedman","year":"2000","journal-title":"Proc AMIA Symp"},{"issue":"3","key":"2020121009244545000_ocaa189-B28","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020121009244545000_ocaa189-B29","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2020121009244545000_ocaa189-B30","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1093\/jamia\/ocx132","article-title":"CLAMP\u2013a toolkit for efficiently building customized clinical natural language processing pipelines","volume":"25","author":"Soysal","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020121009244545000_ocaa189-B31","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1136\/jamia.2010.003939","article-title":"High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge","volume":"17","author":"Patrick","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020121009244545000_ocaa189-B32","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1136\/amiajnl-2012-001607","article-title":"An end-to-end system to identify temporal relation in discharge summaries: 2012 i2b2 challenge","volume":"20","author":"Xu","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244545000_ocaa189-B33","first-page":"54","author":"Pradhan"},{"key":"2020121009244545000_ocaa189-B34","first-page":"303","author":"Elhadad"},{"key":"2020121009244545000_ocaa189-B35","first-page":"1052","author":"Bethard"},{"key":"2020121009244545000_ocaa189-B36","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1007\/978-3-642-40802-1_24","volume-title":"Information Access Evaluation. Multilinguality, Multimodality, and Visualization","author":"Suominen","year":"2013"},{"key":"2020121009244545000_ocaa189-B37","first-page":"172","volume-title":"Information Access Evaluation. Multilinguality, Multimodality, and Interaction","author":"Kelly","year":"2014"},{"key":"2020121009244545000_ocaa189-B38","first-page":"557","volume-title":"J Am Med Inform Assoc","author":"deBruijn","year":"2011"},{"key":"2020121009244545000_ocaa189-B39","first-page":"802","author":"Zhang"},{"key":"2020121009244545000_ocaa189-B40","first-page":"311","author":"Xu"},{"issue":"1","key":"2020121009244545000_ocaa189-B41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/jamia\/ocz206","article-title":"Advancing the state of the art in automatic extraction of adverse drug events from narratives","volume":"27","author":"Uzuner","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244545000_ocaa189-B42","first-page":"467","article-title":"Class-based N-gram Models of Natural Language","volume":"18","author":"Brown","year":"1992","journal-title":"Comput Linguist"},{"key":"2020121009244545000_ocaa189-B43","first-page":"10","author":"Kanerva"},{"key":"2020121009244545000_ocaa189-B44","first-page":"1","author":"Sahlgren"},{"key":"2020121009244545000_ocaa189-B45","first-page":"13","author":"Tang"},{"key":"2020121009244545000_ocaa189-B46","first-page":"1326","article-title":"A study of neural word embeddings for named entity recognition in clinical text","volume":"2015","author":"Wu","year":"2015","journal-title":"AMIA Ann Symp Proc"},{"issue":"8","key":"2020121009244545000_ocaa189-B47","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: a review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2020121009244545000_ocaa189-B48","first-page":"3111","author":"Mikolov","year":"2013"},{"key":"2020121009244545000_ocaa189-B49","first-page":"1532","author":"Pennington"},{"key":"2020121009244545000_ocaa189-B50","author":"Joulin","year":"2016"},{"key":"2020121009244545000_ocaa189-B51","author":"Lample","year":"2016"},{"key":"2020121009244545000_ocaa189-B52","first-page":"1110","article-title":"Combine factual medical knowledge and distributed word representation to improve clinical named entity recognition","volume":"2018","author":"Wu","year":"2018","journal-title":"AMIA Annu Symp Proc"},{"key":"2020121009244545000_ocaa189-B53","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/3-540-33486-6_6","volume-title":"Innovations in Machine Learning: Theory and Applications","author":"Bengio","year":"2006"},{"key":"2020121009244545000_ocaa189-B54","volume-title":"arXiv:170603762 [cs]","author":"Vaswani"},{"issue":"1","key":"2020121009244545000_ocaa189-B55","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/aris.1440390108","article-title":"Statistical language modeling for information retrieval","volume":"39","author":"Liu","year":"2006","journal-title":"Ann Rev Info Sci Technol"},{"issue":"4","key":"2020121009244545000_ocaa189-B56","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1177\/107769905303000401","article-title":"Cloze procedure: a new tool for measuring readability","volume":"30","author":"Taylor","year":"1953","journal-title":"Journalism Quarterly"},{"key":"2020121009244545000_ocaa189-B57","author":"Yan"},{"key":"2020121009244545000_ocaa189-B58","volume-title":"arXiv:180407461 [cs]","author":"Wang"},{"key":"2020121009244545000_ocaa189-B59","volume-title":"arXiv:170405426 [cs]","author":"Williams"},{"key":"2020121009244545000_ocaa189-B60","volume-title":"arXiv:160605250 [cs]","author":"Rajpurkar"},{"key":"2020121009244545000_ocaa189-B61","volume-title":"arXiv:180410959 [cs]","author":"Kudo"},{"key":"2020121009244545000_ocaa189-B62","author":"Wu","year":"2016"},{"key":"2020121009244545000_ocaa189-B63","author":"Sennrich","year":"2015"},{"key":"2020121009244545000_ocaa189-B64","author":"Kudo"},{"issue":"1","key":"2020121009244545000_ocaa189-B65","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2020121009244545000_ocaa189-B66","first-page":"8024","volume-title":"Advances in Neural Information Processing Systems 32","author":"Paszke","year":"2019"},{"issue":"1","key":"2020121009244545000_ocaa189-B67","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1093\/jamia\/ocz144","article-title":"Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting","volume":"27","author":"Yang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"S1","key":"2020121009244545000_ocaa189-B68","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1186\/s12911-020-1045-z","article-title":"Distributed representation and one-hot representation fusion with gated network for clinical semantic textual similarity","volume":"20","author":"Xiong","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"issue":"12","key":"2020121009244545000_ocaa189-B69","doi-asserted-by":"crossref","first-page":"1632","DOI":"10.1093\/jamia\/ocz164","article-title":"Traditional Chinese medicine clinical records classification with BERT and domain specific corpora","volume":"26","author":"Yao","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2020121009244545000_ocaa189-B70","first-page":"1236","article-title":"Relation extraction from clinical narratives using pre-trained language models","volume":"2019","author":"Wei","year":"2019","journal-title":"AMIA Annu Symp Proc"},{"key":"2020121009244545000_ocaa189-B71","doi-asserted-by":"crossref","first-page":"103382","DOI":"10.1016\/j.jbi.2020.103382","article-title":"Multiple features for clinical relation extraction: a machine learning approach","volume":"103","author":"Alimova","year":"2020","journal-title":"J Biomed Inform"},{"key":"2020121009244545000_ocaa189-B72","author":"Schmidt"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/12\/1935\/34838661\/ocaa189.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/12\/1935\/34838661\/ocaa189.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,10]],"date-time":"2020-12-10T14:42:44Z","timestamp":1607611364000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/12\/1935\/5943218"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,29]]},"references-count":72,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,10,29]]},"published-print":{"date-parts":[[2020,12,9]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa189","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,12]]},"published":{"date-parts":[[2020,10,29]]}}}