{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T01:35:17Z","timestamp":1781055317458,"version":"3.54.1"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2019,7,2]],"date-time":"2019-07-02T00:00:00Z","timestamp":1562025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R00LM012104"],"award-info":[{"award-number":["R00LM012104"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM010681"],"award-info":[{"award-number":["R01LM010681"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["U24CA194215"],"award-info":[{"award-number":["U24CA194215"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["RP170668"],"award-info":[{"award-number":["RP170668"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["RR180012"],"award-info":[{"award-number":["RR180012"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Neural network\u2013based representations (\u201cembeddings\u201d) have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (eg, ELMo, BERT) have further pushed the state of the art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText).<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>Both off-the-shelf, open-domain embeddings and pretrained clinical embeddings from MIMIC-III (Medical Information Mart for Intensive Care III) are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings and compare these on 4 concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pretraining time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Contextual embeddings pretrained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate that contextual embeddings encode valuable semantic information not accounted for in traditional word representations.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocz096","type":"journal-article","created":{"date-parts":[[2019,5,24]],"date-time":"2019-05-24T19:11:29Z","timestamp":1558725089000},"page":"1297-1304","source":"Crossref","is-referenced-by-count":236,"title":["Enhancing clinical concept extraction with contextual embeddings"],"prefix":"10.1093","volume":"26","author":[{"given":"Yuqi","family":"Si","sequence":"first","affiliation":[{"name":"School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingqi","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6525-5213","authenticated-orcid":false,"given":"Kirk","family":"Roberts","sequence":"additional","affiliation":[{"name":"School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, Texas, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,7,2]]},"reference":[{"key":"2020110613095483300_ocz096-B1","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1472-6947-13-S1-S1","article-title":"Recognizing clinical entities in hospital discharge summaries using Structural Support Vector Machines with word representation features","volume":"13 (suppl 1)","author":"Tang","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"key":"2020110613095483300_ocz096-B2","first-page":"1937","author":"Kundeti","year":"2016"},{"key":"2020110613095483300_ocz096-B3","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.jbi.2017.11.007","article-title":"Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition","volume":"76","author":"Unanue","year":"2017","journal-title":"J Biomed Inform"},{"key":"2020110613095483300_ocz096-B4","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"issue":"5","key":"2020110613095483300_ocz096-B5","doi-asserted-by":"crossref","first-page":"594","DOI":"10.1136\/amiajnl-2011-000153","article-title":"Automatic extraction of relations between medical concepts in clinical texts","volume":"18","author":"Rink","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613095483300_ocz096-B6","first-page":"1524","article-title":"A Frame-Based NLP System for Cancer-Related Information Extraction","volume":"2018","author":"Si","year":"2018","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613095483300_ocz096-B7","first-page":"28","author":"Lee","year":"2011"},{"key":"2020110613095483300_ocz096-B8","first-page":"1564\u201372.","article-title":"Extracting and integrating data from entire electronic health records for detecting colorectal cancer cases","volume":"2011","author":"Xu","year":"2011","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613095483300_ocz096-B9","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.jbi.2018.10.005","article-title":"Using clinical Natural Language Processing for health outcomes research: overview and actionable suggestions for future advances","volume":"88","author":"Velupillai","year":"2018","journal-title":"J Biomed Inform"},{"issue":"5","key":"2020110613095483300_ocz096-B10","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110613095483300_ocz096-B11","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613095483300_ocz096-B12","doi-asserted-by":"crossref","first-page":"S11","DOI":"10.1016\/j.jbi.2015.06.007","article-title":"Automated systems for the de-identification of longitudinal clinical narratives: overview of 2014 i2b2\/UTHealth shared task Track 1","volume":"58","author":"Stubbs","year":"2015","journal-title":"J Biomed Inform"},{"key":"2020110613095483300_ocz096-B13","first-page":"212","author":"Suominen","year":"2013"},{"key":"2020110613095483300_ocz096-B14","first-page":"172","author":"Kelly","year":"2014"},{"key":"2020110613095483300_ocz096-B15","first-page":"54","author":"Pradhan","year":"2014"},{"key":"2020110613095483300_ocz096-B16","first-page":"303","author":"Elhadad","year":"2015"},{"key":"2020110613095483300_ocz096-B17","first-page":"1052","author":"Bethard","year":"2016"},{"key":"2020110613095483300_ocz096-B18","first-page":"3511","author":"Shen","year":"2017"},{"issue":"14","key":"2020110613095483300_ocz096-B19","doi-asserted-by":"crossref","first-page":"i37","DOI":"10.1093\/bioinformatics\/btx228","article-title":"Deep learning with word embeddings improves biomedical named entity recognition","volume":"33","author":"Habibi","year":"2017","journal-title":"Bioinformatics"},{"issue":"5","key":"2020110613095483300_ocz096-B20","first-page":"321","article-title":"Application of word embeddings in biomedical named entity recognition tasks","volume":"13","author":"Chang","year":"2015","journal-title":"J Digit Inf Manag"},{"key":"2020110613095483300_ocz096-B21","first-page":"1326","article-title":"A study of neural word embeddings for named entity recognition in clinical text","volume":"2015","author":"Wu","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613095483300_ocz096-B22","first-page":"7","author":"Florez","year":"2018"},{"key":"2020110613095483300_ocz096-B23","author":"Peters"},{"key":"2020110613095483300_ocz096-B24","author":"Devlin"},{"key":"2020110613095483300_ocz096-B25","author":"Seo"},{"key":"2020110613095483300_ocz096-B26","first-page":"3111","author":"Mikolov","year":"2013"},{"key":"2020110613095483300_ocz096-B27","first-page":"1532","author":"Pennington","year":"2014"},{"key":"2020110613095483300_ocz096-B28","author":"Bojanowski"},{"key":"2020110613095483300_ocz096-B29","first-page":"54","author":"Roberts","year":"2016"},{"key":"2020110613095483300_ocz096-B30","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.jbi.2018.09.008","article-title":"A comparison of word embeddings for the biomedical natural language processing","volume":"87","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"2020110613095483300_ocz096-B31"},{"issue":"5","key":"2020110613095483300_ocz096-B32","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1136\/amiajnl-2011-000150","article-title":"Machine-learned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010","volume":"18","author":"de Bruijn","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613095483300_ocz096-B33","author":"Chalapathy"},{"key":"2020110613095483300_ocz096-B34","author":"Zhu"},{"key":"2020110613095483300_ocz096-B35","author":"Lee","year":"2019"},{"key":"2020110613095483300_ocz096-B36","author":"Alsentzer"},{"key":"2020110613095483300_ocz096-B37","author":"Huang","year":"2019"},{"key":"2020110613095483300_ocz096-B38","doi-asserted-by":"crossref","first-page":"160035.","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2020110613095483300_ocz096-B39","first-page":"1184","article-title":"Recognizing disjoint clinical concepts in clinical text using machine learning-based methods","volume":"2015","author":"Tang","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613095483300_ocz096-B40","author":"Lample"},{"key":"2020110613095483300_ocz096-B41","first-page":"265","article-title":"Tensorflow: a system for large-scale machine learning","author":"Abadi","year":"2016"},{"key":"2020110613095483300_ocz096-B42","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1186\/s12911-017-0468-7","article-title":"Entity recognition from clinical texts via recurrent neural network","volume":"17","author":"Liu","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"key":"2020110613095483300_ocz096-B43","first-page":"802","author":"Zhang","year":"2014"},{"key":"2020110613095483300_ocz096-B44","first-page":"572","article-title":"Semantic similarity and relatedness between clinical terms: an experimental study","volume":"2010","author":"Pakhomov","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613095483300_ocz096-B45","doi-asserted-by":"crossref","first-page":"3635","DOI":"10.1093\/bioinformatics\/btw529","article-title":"Corpus domain effects on distributional semantic modeling of medical terms","volume":"32","author":"Pakhomov","year":"2016","journal-title":"Bioinformatics"},{"key":"2020110613095483300_ocz096-B46","first-page":"5149","author":"Schuster","year":"2012"},{"key":"2020110613095483300_ocz096-B47","first-page":"18","volume-title":"The Specialist Lexicon","author":"Browne","year":"2000"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/11\/1297\/34152015\/ocz096.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/26\/11\/1297\/34152015\/ocz096.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,16]],"date-time":"2023-09-16T22:05:59Z","timestamp":1694901959000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/26\/11\/1297\/5527248"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,2]]},"references-count":47,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2019,7,2]]},"published-print":{"date-parts":[[2019,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz096","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,11]]},"published":{"date-parts":[[2019,7,2]]}}}