{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T01:48:24Z","timestamp":1780537704959,"version":"3.54.1"},"reference-count":65,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2020,9,24]],"date-time":"2020-09-24T00:00:00Z","timestamp":1600905600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"National Library of Medicine of the National Institutes of Health","award":["R13LM013127"],"award-info":[{"award-number":["R13LM013127"]}]},{"name":"National Library of Medicine of the National Institutes of Health","award":["R13LM011411"],"award-info":[{"award-number":["R13LM011411"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>The n2c2\/UMass Lowell spin-off shared task focused on medical concept normalization (MCN) in clinical records. This task aimed to assess state-of-the-art methods for matching salient medical concepts from clinical records to a controlled vocabulary. We describe the task and the dataset used, compare the participating systems, and identify the strengths and limitations of the current approaches and directions for future research.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>Participating teams were asked to link preselected text spans in discharge summaries (henceforth referred to as concept mentions) to the corresponding concepts in the SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and RxNorm vocabularies from the Unified Medical Language System. The shared task used the MCN corpus created by the organizers, which maps all mentions of problems, treatments, and tests in the 2010 i2b2\/VA challenge data to the Unified Medical Language System concepts. Submitted systems represented 4 broad categories of approaches: cascading dictionary matching, cosine distance, deep learning, and retrieve-and-rank systems. Disambiguation modules were common across all approaches.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>A total of 33 teams participated in the shared task. The best-performing team achieved an accuracy of 0.8526. The median and mean performances among all teams were 0.7733 and 0.7426, respectively.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Overall performance among the top 10 teams was high. However, particularly challenging for all teams were mentions requiring disambiguation of misspelled words, acronyms, abbreviations, and mentions with more than 1 possible semantic type. Complex mentions of long, multiword terms were also challenging and, in the future, will require better methods for learning contextualized representations of concept mentions and better use of domain knowledge.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocaa106","type":"journal-article","created":{"date-parts":[[2020,5,15]],"date-time":"2020-05-15T03:35:05Z","timestamp":1589513705000},"page":"1529-e1","source":"Crossref","is-referenced-by-count":37,"title":["The 2019 n2c2\/UMass Lowell shared task on clinical concept normalization"],"prefix":"10.1093","volume":"27","author":[{"given":"Yen-Fu","family":"Luo","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Massachusetts Lowell, Lowell, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7246-1151","authenticated-orcid":false,"given":"Sam","family":"Henry","sequence":"additional","affiliation":[{"name":"Department of Information Sciences and Technology, George Mason University, Fairfax, Virginia, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanshan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Feichen","family":"Shen","sequence":"additional","affiliation":[{"name":"Department of Health Sciences Research, Mayo Clinic, Rochester, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8011-9850","authenticated-orcid":false,"given":"Ozlem","family":"Uzuner","sequence":"additional","affiliation":[{"name":"Department of Information Sciences and Technology, George Mason University, Fairfax, Virginia, USA"},{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA"},{"name":"Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna","family":"Rumshisky","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Massachusetts Lowell, Lowell, Massachusetts, USA"},{"name":"Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,9,24]]},"reference":[{"key":"2021081413411308300_ocaa106-B1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-43742-2","volume-title":"Secondary Analysis of Electronic Health Records","author":"Critical Data","year":"2016"},{"key":"2021081413411308300_ocaa106-B2","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-78503-5","volume-title":"Clinical Text Mining: Secondary Use of Electronic Patient Records","author":"Dalianis","year":"2018"},{"issue":"5","key":"2021081413411308300_ocaa106-B3","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","article-title":"Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis","volume":"22","author":"Shickel","year":"2017","journal-title":"IEEE J Biomed Health Inform"},{"key":"2021081413411308300_ocaa106-B4","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/978-3-030-33966-1_8","volume-title":"Deep Learning Techniques for Biomedical and Health Informatics","author":"Singh Gangwar","year":"2020"},{"issue":"1","key":"2021081413411308300_ocaa106-B5","doi-asserted-by":"crossref","first-page":"26094","DOI":"10.1038\/srep26094","article-title":"Deep patient: an unsupervised representation to predict the future of patients from the electronic health records","volume":"6","author":"Miotto","year":"2016","journal-title":"Sci Rep"},{"issue":"13","key":"2021081413411308300_ocaa106-B6","doi-asserted-by":"crossref","first-page":"2221","DOI":"10.1017\/S0033291719002320","article-title":"Characterizing DSM-5 and ICD-11 personality disorder features in psychiatric inpatients at scale using electronic health records","volume":"50","author":"Barroilhet","year":"2020","journal-title":"Psychol Med"},{"issue":"5","key":"2021081413411308300_ocaa106-B7","doi-asserted-by":"crossref","first-page":"e0154515","DOI":"10.1371\/journal.pone.0154515","article-title":"Defining disease phenotypes in primary care electronic health records by a machine learning approach: a case study in identifying rheumatoid arthritis","volume":"11","author":"Zhou","year":"2016","journal-title":"PLoS One"},{"key":"2021081413411308300_ocaa106-B8","doi-asserted-by":"crossref","first-page":"105055","DOI":"10.1016\/j.cmpb.2019.105055","article-title":"Predicting the onset of type 2 diabetes using wide and deep learning with electronic health records","volume":"182","author":"Nguyen","year":"2019","journal-title":"Comput Methods Programs Biomed"},{"issue":"1","key":"2021081413411308300_ocaa106-B9","doi-asserted-by":"crossref","first-page":"e22","DOI":"10.2196\/jmir.9268","article-title":"Prediction of incident hypertension within the next year: prospective study using statewide electronic health records and machine learning","volume":"20","author":"Ye","year":"2018","journal-title":"J Med Internet Res"},{"key":"2021081413411308300_ocaa106-B10","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.ijmedinf.2016.09.014","article-title":"A machine learning-based framework to identify type 2 diabetes through electronic health records","volume":"97","author":"Zheng","year":"2017","journal-title":"Int J Med Inform"},{"issue":"10","key":"2021081413411308300_ocaa106-B11","doi-asserted-by":"crossref","first-page":"e921","DOI":"10.1038\/tp.2015.182","article-title":"Predicting early psychiatric readmission with natural language processing of narrative discharge summaries","volume":"6","author":"Rumshisky","year":"2016","journal-title":"Transl Psychiatry"},{"key":"2021081413411308300_ocaa106-B12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.compbiomed.2017.12.026","article-title":"Prediction of venous thromboembolism using semantic and sentiment analyses of clinical narratives","volume":"94","author":"Sabra","year":"2018","journal-title":"Comput Biol Med"},{"key":"2021081413411308300_ocaa106-B13","first-page":"6103","author":"Liu","year":"2019"},{"key":"2021081413411308300_ocaa106-B14","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.jbi.2017.06.019","article-title":"Automatic prediction of coronary artery disease from clinical narratives","volume":"72","author":"Buchan","year":"2017","journal-title":"J Biomed Inform"},{"key":"2021081413411308300_ocaa106-B15","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): integrating biomedical terminology","volume":"32(suppl_1","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2021081413411308300_ocaa106-B16","article-title":"g","author":"Devlin","year":"2019"},{"issue":"4","key":"2021081413411308300_ocaa106-B17","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2021081413411308300_ocaa106-B18","first-page":"72","author":"Alsentzer","year":"2019"},{"key":"2021081413411308300_ocaa106-B19","doi-asserted-by":"crossref","first-page":"103132","DOI":"10.1016\/j.jbi.2019.103132","article-title":"MCN: a comprehensive corpus for medical concept normalization","volume":"92","author":"Luo","year":"2019","journal-title":"J Biomed Inform"},{"issue":"5","key":"2021081413411308300_ocaa106-B20","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/va challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2021081413411308300_ocaa106-B21","first-page":"640","author":"Spackman","year":"1997"},{"issue":"5","key":"2021081413411308300_ocaa106-B22","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1109\/MITP.2005.122","article-title":"Rxnorm: prescription for electronic drug information exchange","volume":"7","author":"Liu","year":"2005","journal-title":"IT Professional"},{"key":"2021081413411308300_ocaa106-B23","first-page":"732","author":"Luo","year":"2019"},{"key":"2021081413411308300_ocaa106-B24","first-page":"17","author":"Aronson","year":"2001"},{"issue":"5","key":"2021081413411308300_ocaa106-B25","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2021081413411308300_ocaa106-B26","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1093\/jamia\/ocx132","article-title":"CLAMP\u2013a toolkit for efficiently building customized clinical natural language processing pipelines","volume":"25","author":"Soysal","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2021081413411308300_ocaa106-B27","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.jbi.2015.07.010","article-title":"Challenges in clinical natural language processing for automated disorder normalization","volume":"57","author":"Leaman","year":"2015","journal-title":"J Biomed Inform"},{"key":"2021081413411308300_ocaa106-B28","first-page":"212","author":"Suominen","year":"2013:"},{"key":"2021081413411308300_ocaa106-B29","first-page":"54","author":"Pradhan","year":"2014:"},{"key":"2021081413411308300_ocaa106-B30","first-page":"303","author":"Elhadad","year":"2015:"},{"issue":"Suppl 1","key":"2021081413411308300_ocaa106-B31","doi-asserted-by":"crossref","first-page":"S11","DOI":"10.1186\/1471-2105-6-S1-S11","article-title":"Overview of BioCreative task 1b: normalized gene lists","volume":"6","author":"Hirschman","year":"2005","journal-title":"BMC Bioinformatics"},{"issue":"Suppl 2","key":"2021081413411308300_ocaa106-B32","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1186\/gb-2008-9-s2-s3","article-title":"Overview of BioCreative II gene normalization","volume":"9","author":"Morgan","year":"2008","journal-title":"Genome Biol"},{"issue":"S8","key":"2021081413411308300_ocaa106-B33","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/1471-2105-12-S8-S2","article-title":"The gene normalization task in BioCreative III","volume":"12","author":"Lu","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2021081413411308300_ocaa106-B34","doi-asserted-by":"crossref","first-page":"baw068","DOI":"10.1093\/database\/baw068","article-title":"BioCreative v CDR task corpus: a resource for chemical disease relation extraction","volume":"2016","author":"Li","year":"2016","journal-title":"Database"},{"key":"2021081413411308300_ocaa106-B35","author":"Roberts","year":"2017"},{"key":"2021081413411308300_ocaa106-B36","author":"Leaman","year":"2009"},{"key":"2021081413411308300_ocaa106-B37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Do\u011fan","year":"2014","journal-title":"J Biomed Inform"},{"issue":"(10","key":"2021081413411308300_ocaa106-B38","first-page":"1239","article-title":"Overview of the second social media mining for health (SMM4H) shared tasks at AMIA 2017","volume":"822","author":"Sarker","year":"2017","journal-title":"Training"},{"key":"2021081413411308300_ocaa106-B39","author":"Limsopatham"},{"key":"2021081413411308300_ocaa106-B40","first-page":"1014","author":"Limsopatham","year":"2016"},{"key":"2021081413411308300_ocaa106-B41","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.jbi.2015.03.010","article-title":"CADEC: a corpus of adverse drug event annotations","volume":"55","author":"Karimi","year":"2015","journal-title":"J Biomed Inform"},{"key":"2021081413411308300_ocaa106-B42","first-page":"D52","article-title":"Entrez gene: gene-centered information at NCBI","volume":"39(suppl_1","author":"Maglott","year":"2010","journal-title":"Nucleic Acids Res"},{"issue":"3","key":"2021081413411308300_ocaa106-B43","first-page":"265","article-title":"Medical subject headings (MeSH)","volume":"88","author":"Lipscomb","year":"2000","journal-title":"Bull Med Libr Assoc"},{"issue":"2","key":"2021081413411308300_ocaa106-B44","doi-asserted-by":"crossref","first-page":"109","DOI":"10.2165\/00002018-199920020-00002","article-title":"The medical dictionary for regulatory activities (MEDDRA)","volume":"20","author":"Brown","year":"1999","journal-title":"Drug Saf"},{"issue":"5","key":"2021081413411308300_ocaa106-B45","doi-asserted-by":"crossref","first-page":"259","DOI":"10.2165\/00124363-200418050-00001","article-title":"Medical dictionary for regulatory activities (MEDDRA)","volume":"18","author":"Fescharek","year":"2004","journal-title":"Int J Pharm Med"},{"key":"2021081413411308300_ocaa106-B46","doi-asserted-by":"crossref","first-page":"bar065","DOI":"10.1093\/database\/bar065","article-title":"Medic: a practical disease vocabulary used at the comparative toxicogenomics database","volume":"2012","author":"Davis","year":"2012","journal-title":"Database (Oxford)"},{"issue":"Database issue","key":"2021081413411308300_ocaa106-B47","doi-asserted-by":"crossref","first-page":"D514","DOI":"10.1093\/nar\/gki033","article-title":"Online Mendelian inheritance in man (OMIM), a knowledgebase of human genes and genetic disorders","volume":"33","author":"Hamosh","year":"2005","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"2021081413411308300_ocaa106-B48","doi-asserted-by":"crossref","first-page":"D1075","DOI":"10.1093\/nar\/gkv1075","article-title":"The sider database of drugs and side effects","volume":"44","author":"Kuhn","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2021081413411308300_ocaa106-B49","author":"NEHTA","year":"2014"},{"key":"2021081413411308300_ocaa106-B50","first-page":"641","author":"Saeed","year":"2002"},{"key":"2021081413411308300_ocaa106-B51","first-page":"129","author":"Stubbs","year":"2011:"},{"key":"2021081413411308300_ocaa106-B52","volume-title":"Computer-Intensive Methods for Testing Hypotheses","author":"Noreen","year":"1989"},{"key":"2021081413411308300_ocaa106-B53","first-page":"947","author":"Yeh","year":"2000:"},{"issue":"1","key":"2021081413411308300_ocaa106-B54","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/jamia\/ocz166","article-title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records","volume":"27","author":"Henry","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2021081413411308300_ocaa106-B55","first-page":"30","author":"Chinchor","year":"1992:"},{"issue":"11","key":"2021081413411308300_ocaa106-B56","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1093\/jamia\/ocz163","article-title":"Cohort selection for clinical trials: n2c2 2018 shared task track 1","volume":"26","author":"Stubbs","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2021081413411308300_ocaa106-B57","doi-asserted-by":"crossref","first-page":"S11","DOI":"10.1016\/j.jbi.2015.06.007","article-title":"Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2\/UTHealth shared task track 1","volume":"58","author":"Stubbs","year":"2015","journal-title":"J Biomed Informatics"},{"issue":"2","key":"2021081413411308300_ocaa106-B58","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1093\/jamia\/ocv108","article-title":"Normalizing clinical terms using learned edit distance patterns","volume":"23","author":"Kate","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2021081413411308300_ocaa106-B59","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1075\/term.00015.tho","article-title":"HYPHEN: a flexible, hybrid method to map phenotype concept mentions to terminological resources","volume":"24","author":"Thompson","year":"2018","journal-title":"Terminology"},{"key":"2021081413411308300_ocaa106-B60","author":"Beltagy","year":"2019"},{"key":"2021081413411308300_ocaa106-B61","first-page":"4690","author":"Deng","year":"2019:"},{"key":"2021081413411308300_ocaa106-B62","first-page":"2623","author":"Akiba","year":"2019:"},{"key":"2021081413411308300_ocaa106-B63","author":"Chen","year":"2020"},{"issue":"3\u20134","key":"2021081413411308300_ocaa106-B64","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1017\/S1351324904003523","article-title":"UIMA: an architectural approach to unstructured information processing in the corporate research environment","volume":"10","author":"Ferrucci","year":"2004","journal-title":"Nat Lang Eng"},{"key":"2021081413411308300_ocaa106-B65","first-page":"1310","author":"Moon","year":"2012:"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/10\/1529\/39739985\/ocaa106.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/10\/1529\/39739985\/ocaa106.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,1]],"date-time":"2023-10-01T03:21:52Z","timestamp":1696130512000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/10\/1529\/5910736"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,24]]},"references-count":65,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2020,9,24]]},"published-print":{"date-parts":[[2020,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa106","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,10]]},"published":{"date-parts":[[2020,9,24]]}}}