{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T08:47:43Z","timestamp":1770972463890,"version":"3.50.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2020,7,28]],"date-time":"2020-07-28T00:00:00Z","timestamp":1595894400000},"content-version":"vor","delay-in-days":1,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM012918"],"award-info":[{"award-number":["R01LM012918"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1228509"],"award-info":[{"award-number":["1228509"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"LLC","award":["DE-AC05-00OR22725"],"award-info":[{"award-number":["DE-AC05-00OR22725"]}]},{"name":"National Institutes of Health, National Science Foundation, UT-Battelle, or the Department of Energy"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Concept normalization, the task of linking phrases in text to concepts in an ontology, is useful for many downstream tasks including relation extraction, information retrieval, etc. We present a generate-and-rank concept normalization system based on our participation in the 2019 National NLP Clinical Challenges Shared Task Track 3 Concept Normalization.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>The shared task provided 13\u00a0609 concept mentions drawn from 100 discharge summaries. We first design a sieve-based system that uses Lucene indices over the training data, Unified Medical Language System (UMLS) preferred terms, and UMLS synonyms to generate a list of possible concepts for each mention. We then design a listwise classifier based on the BERT (Bidirectional Encoder Representations from Transformers) neural network to rank the candidate concepts, integrating UMLS semantic types through a regularizer.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Our generate-and-rank system was third of 33 in the competition, outperforming the candidate generator alone (81.66% vs 79.44%) and the previous state of the art (76.35%). During postevaluation, the model\u2019s accuracy was increased to 83.56% via improvements to how training data are generated from UMLS and incorporation of our UMLS semantic type regularizer.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Analysis of the model shows that prioritizing UMLS preferred terms yields better performance, that the UMLS semantic type regularizer results in qualitatively better concept predictions, and that the model performs well even on concepts not seen during training.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Our generate-and-rank framework for UMLS concept normalization integrates key UMLS features like preferred terms and semantic types with a neural network\u2013based ranking model to accurately link phrases in text to UMLS concepts.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocaa080","type":"journal-article","created":{"date-parts":[[2020,4,28]],"date-time":"2020-04-28T03:20:41Z","timestamp":1588044041000},"page":"1510-1519","source":"Crossref","is-referenced-by-count":23,"title":["Unified Medical Language System resources improve sieve-based generation and Bidirectional Encoder Representations from Transformers (BERT)\u2013based ranking for concept normalization"],"prefix":"10.1093","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0828-1102","authenticated-orcid":false,"given":"Dongfang","family":"Xu","sequence":"first","affiliation":[{"name":"School of Information, University of Arizona, Tucson, Arizona, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Manoj","family":"Gopale","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Arizona, Tucson, Arizona, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7267-5232","authenticated-orcid":false,"given":"Jiacheng","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Arizona, Tucson, Arizona, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kris","family":"Brown","sequence":"additional","affiliation":[{"name":"National Center for Computational Sciences, Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edmon","family":"Begoli","sequence":"additional","affiliation":[{"name":"National Center for Computational Sciences, Oak Ridge National Laboratory, Oak Ridge, Tennessee, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9560-6491","authenticated-orcid":false,"given":"Steven","family":"Bethard","sequence":"additional","affiliation":[{"name":"School of Information, University of Arizona, Tucson, Arizona, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2020,7,27]]},"reference":[{"key":"2020110613082441200_ocaa080-B1","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.ymeth.2015.01.015","article-title":"Application of text mining in the biomedical domain","volume":"74","author":"Fleuren","year":"2015","journal-title":"Methods"},{"issue":"1","key":"2020110613082441200_ocaa080-B2","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1093\/bib\/bbv087","article-title":"Recent advances and emerging applications in text and data mining for biomedical discovery","volume":"17","author":"Gonzalez","year":"2016","journal-title":"Brief Bioinform"},{"issue":"1","key":"2020110613082441200_ocaa080-B3","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s40264-018-0766-8","article-title":"Towards drug safety surveillance and pharmacovigilance: current progress in detecting medication and adverse drug events from electronic health records","volume":"42","author":"Liu","year":"2019","journal-title":"Drug Saf"},{"issue":"10","key":"2020110613082441200_ocaa080-B4","doi-asserted-by":"crossref","first-page":"e921","DOI":"10.1038\/tp.2015.182","article-title":"Predicting early psychiatric readmission with natural language processing of narrative discharge summaries","volume":"6","author":"Rumshisky","year":"2016","journal-title":"Transl Psychiatry"},{"issue":"90001","key":"2020110613082441200_ocaa080-B5","doi-asserted-by":"crossref","first-page":"267D","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2020110613082441200_ocaa080-B6","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1007\/978-3-319-11382-1_17","article-title":"Overview of the ShARe\/CLEF eHealth evaluation lab 2013","volume":"8685","author":"Kelly","year":"2014","journal-title":"Lect Notes Comput Sci"},{"key":"2020110613082441200_ocaa080-B7","first-page":"54","author":"Pradhan","year":"2014"},{"key":"2020110613082441200_ocaa080-B8","first-page":"21","author":"Weissenbacher","year":"2019"},{"key":"2020110613082441200_ocaa080-B9","first-page":"43","article-title":"Overview of the Second Social Media Mining for Health (SMM4H) Shared Tasks at AMIA 2017","author":"Sarker","year":"1996","journal-title":"CEUR Workshop Proceedings 2017"},{"key":"2020110613082441200_ocaa080-B10","doi-asserted-by":"crossref","first-page":"103132","DOI":"10.1016\/j.jbi.2019.103132","article-title":"MCN: a comprehensive corpus for medical concept normalization","volume":"92","author":"Luo","year":"2019","journal-title":"J Biomed Inform"},{"key":"2020110613082441200_ocaa080-B11","doi-asserted-by":"crossref","first-page":"baw091","DOI":"10.1093\/database\/baw091","article-title":"AuDis: an automatic CRF-enhanced disease normalization in biomedical text","volume":"2016","author":"Lee","year":"2016","journal-title":"Database (Oxford)"},{"key":"2020110613082441200_ocaa080-B12","first-page":"406","author":"Martins","year":"2015"},{"issue":"2","key":"2020110613082441200_ocaa080-B13","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1093\/jamia\/ocv108","article-title":"Normalizing clinical terms using learned edit distance patterns","volume":"23","author":"Kate","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613082441200_ocaa080-B14","first-page":"17","article-title":"Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap Program","volume":"2001","author":"Aronson","year":"2001","journal-title":"Proc AMIA Symp"},{"issue":"5","key":"2020110613082441200_ocaa080-B15","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Chute","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613082441200_ocaa080-B16","doi-asserted-by":"crossref","first-page":"S189","DOI":"10.1016\/j.jbi.2015.07.008","article-title":"Ease of adoption of clinical natural language processing software: an evaluation of five systems","volume":"58","author":"Zheng","year":"2015","journal-title":"J Biomed Inform"},{"issue":"5","key":"2020110613082441200_ocaa080-B17","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1136\/amiajnl-2011-000465","article-title":"Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions","volume":"18","author":"Chapman","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613082441200_ocaa080-B18","first-page":"297","author":"D\u2019Souza","year":"2015"},{"key":"2020110613082441200_ocaa080-B19","doi-asserted-by":"crossref","first-page":"baw112","DOI":"10.1093\/database\/baw112","article-title":"Improving the dictionary lookup approach for disease normalization using enhanced dictionary and query expansion","volume":"2016","author":"Jonnagaddala","year":"2016","journal-title":"Database (Oxford)"},{"key":"2020110613082441200_ocaa080-B20","doi-asserted-by":"publisher","author":"Stevenson","year":"2009","DOI":"10.3115\/1572364.1572374"},{"issue":"6","key":"2020110613082441200_ocaa080-B21","doi-asserted-by":"crossref","first-page":"1088","DOI":"10.1016\/j.jbi.2008.02.003","article-title":"Word sense disambiguation across two domains: Biomedical literature and clinical notes","volume":"41","author":"Savova","year":"2008","journal-title":"J Biomed Inform"},{"key":"2020110613082441200_ocaa080-B22","first-page":"1014","author":"Limsopatham","year":"2016"},{"key":"2020110613082441200_ocaa080-B23","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.jbi.2017.08.001","article-title":"Word embeddings and recurrent neural networks based on Long-Short Term Memory nodes in supervised biomedical word sense disambiguation","volume":"73","author":"Jimeno Yepes","year":"2017","journal-title":"J Biomed Inform"},{"key":"2020110613082441200_ocaa080-B24","first-page":"8","article-title":"Word sense disambiguation of medical terms via recurrent convolutional neural networks","volume":"236","author":"Festag","year":"2017","journal-title":"Stud Health Technol Inform"},{"key":"2020110613082441200_ocaa080-B25","first-page":"462","author":"Lee","year":"2017"},{"key":"2020110613082441200_ocaa080-B26","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.jbi.2018.06.006","article-title":"Medical concept normalization in social media posts with recurrent neural networks","volume":"84","author":"Tutubalina","year":"2018","journal-title":"J Biomed Inform"},{"issue":"3","key":"2020110613082441200_ocaa080-B27","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1007\/s11063-018-9873-x","article-title":"Multi-task character-level attentional networks for medical concept normalization","volume":"49","author":"Niu","year":"2019","journal-title":"Neural Process Lett"},{"key":"2020110613082441200_ocaa080-B28","first-page":"732","article-title":"A hybrid normalization method for medical concepts in clinical narrative using semantic matching","volume":"2019","author":"Luo","year":"2019","journal-title":"AMIA Jt Summits on Transl Sci Proc"},{"key":"2020110613082441200_ocaa080-B29","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1007\/978-3-319-73618-1_13","article-title":"A deep learning way for disease name representation and normalization","volume":"10619","author":"Liu","year":"2018","journal-title":"Lect Notes Comput Sci"},{"issue":"22","key":"2020110613082441200_ocaa080-B30","doi-asserted-by":"crossref","first-page":"2909","DOI":"10.1093\/bioinformatics\/btt474","article-title":"DNorm: Disease name normalization with pairwise learning to rank","volume":"29","author":"Leaman","year":"2013","journal-title":"Bioinformatics"},{"issue":"S11","key":"2020110613082441200_ocaa080-B31","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1186\/s12859-017-1805-7","article-title":"CNN-based ranking for biomedical entity normalization","volume":"18","author":"Li","year":"2017","journal-title":"BMC Bioinformatics"},{"key":"2020110613082441200_ocaa080-B32","author":"Nguy\u1ec5n","year":"2018"},{"key":"2020110613082441200_ocaa080-B33","first-page":"97","author":"Murty","year":"2018"},{"key":"2020110613082441200_ocaa080-B34","first-page":"129","article-title":"Learning to rank: from pairwise approach to listwise approach","volume":"227","author":"Cao","year":"2007","journal-title":"ACM Int Conf Proc Ser"},{"key":"2020110613082441200_ocaa080-B35","first-page":"1192","author":"Xia","year":"2008"},{"key":"2020110613082441200_ocaa080-B36","first-page":"2227","author":"Peters","year":"2018"},{"key":"2020110613082441200_ocaa080-B37","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2020110613082441200_ocaa080-B38","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2020110613082441200_ocaa080-B39","first-page":"72","author":"Alsentzer","year":"2019"},{"key":"2020110613082441200_ocaa080-B40","author":"Huang"},{"issue":"11","key":"2020110613082441200_ocaa080-B41","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1093\/jamia\/ocz096","article-title":"Enhancing clinical concept extraction with contextual embeddings","volume":"26","author":"Si","year":"2019","journal-title":"J Am Med Informatics Assoc"},{"key":"2020110613082441200_ocaa080-B42","first-page":"58","author":"Peng","year":"2019"},{"key":"2020110613082441200_ocaa080-B43","doi-asserted-by":"crossref","first-page":"e14830","DOI":"10.2196\/14830","article-title":"Fine-tuning bidirectional encoder representations from transformers (BERT)\u2013based models on large-scale electronic health record notes: an empirical study","volume":"7","author":"Li","year":"2019","journal-title":"JMIR Med Inform"},{"key":"2020110613082441200_ocaa080-B44","author":"Ji","year":"2019"},{"key":"2020110613082441200_ocaa080-B45","first-page":"393","author":"Miftahutdinov","year":"2019"},{"key":"2020110613082441200_ocaa080-B46","first-page":"1236","article-title":"Relation extraction from clinical narratives using pre-trained language models","volume":"2019","author":"Wei","year":"2019","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613082441200_ocaa080-B47","first-page":"65","author":"Lin","year":"2019"},{"issue":"5","key":"2020110613082441200_ocaa080-B48","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Informatics Assoc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/10\/1510\/34152904\/ocaa080.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/10\/1510\/34152904\/ocaa080.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,30]],"date-time":"2023-09-30T11:16:47Z","timestamp":1696072607000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/10\/1510\/5876963"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,27]]},"references-count":48,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2020,7,27]]},"published-print":{"date-parts":[[2020,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa080","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,10]]},"published":{"date-parts":[[2020,7,27]]}}}