{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T19:27:50Z","timestamp":1772134070788,"version":"3.50.1"},"reference-count":63,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T00:00:00Z","timestamp":1673568000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Objective<\/jats:title><jats:p>The adoption of electronic health records (EHRs) has produced enormous amounts of data, creating research opportunities in clinical data sciences. Several concept recognition systems have been developed to facilitate clinical information extraction from these data. While studies exist that compare the performance of many concept recognition systems, they are typically developed internally and may be biased due to different internal implementations, parameters used, and limited number of systems included in the evaluations. The goal of this research is to evaluate the performance of existing systems to retrieve relevant clinical concepts from EHRs.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We investigated six concept recognition systems, including CLAMP, cTAKES, MetaMap, NCBO Annotator, QuickUMLS, and ScispaCy. Clinical concepts extracted included procedures, disorders, medications, and anatomical location. The system performance was evaluated on two datasets: the 2010 i2b2 and the MIMIC-III. Additionally, we assessed the performance of these systems in five challenging situations, including negation, severity, abbreviation, ambiguity, and misspelling.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>For clinical concept extraction, CLAMP achieved the best performance on exact and inexact matching, with an F-score of 0.70 and 0.94, respectively, on i2b2; and 0.39 and 0.50, respectively, on MIMIC-III. Across the five challenging situations, ScispaCy excelled in extracting abbreviation information (F-score: 0.86) followed by NCBO Annotator (F-score: 0.79). CLAMP outperformed in extracting severity terms (F-score 0.73) followed by NCBO Annotator (F-score: 0.68). CLAMP outperformed other systems in extracting negated concepts (F-score 0.63).<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Several concept recognition systems exist to extract clinical information from unstructured data. This study provides an external evaluation by end-users of six commonly used systems across different extraction tasks. Our findings suggest that CLAMP provides the most comprehensive set of annotations for clinical concept extraction tasks and associated challenges. Comparing standard extraction tasks across systems provides guidance to other clinical researchers when selecting a concept recognition system relevant to their clinical information extraction task.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2022.1051724","type":"journal-article","created":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T05:22:51Z","timestamp":1673587371000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Clinical concept recognition: Evaluation of existing systems on EHRs"],"prefix":"10.3389","volume":"5","author":[{"given":"Juan Antonio","family":"Lossio-Ventura","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ran","family":"Sun","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastien","family":"Boussard","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tina","family":"Hernandez-Boussard","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,1,13]]},"reference":[{"key":"B1","unstructured":"Clinical Text Analysis Knowledge Extraction System2021"},{"key":"B2","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B3","doi-asserted-by":"publisher","first-page":"66","DOI":"10.3389\/fmed.2019.00066","article-title":"The revival of the notes field: leveraging the unstructured content in electronic health records","volume":"6","author":"Assale","year":"2019","journal-title":"Front. Med"},{"key":"B4","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1146\/annurev-biodatasci-080917-013315","article-title":"Advances in electronic phenotyping: from rule-based definitions to machine learning models","volume":"1","author":"Banda","year":"2018","journal-title":"Annu. Rev. Biomed. Data Sci"},{"key":"B5","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1002\/lrh2.10237","article-title":"Phenotyping severity of patient-centered outcomes using clinical notes: a prostate cancer use case","volume":"4","author":"Bozkurt","year":"2020","journal-title":"Learn. Health Syst"},{"key":"B7","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","year":"2001","journal-title":"J. Biomed. Inform"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103381","article-title":"Combinatorial feature embedding based on CNN and LSTM for biomedical named entity recognition","volume":"103","author":"Cho","year":"2020","journal-title":"J. Biomed. Inform"},{"key":"B9","unstructured":"Natural Language Processing (NLP) Software2021"},{"key":"B10","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1093\/jamia\/ocw177","article-title":"MetaMap Lite: an evaluation of a new Java implementation of MetaMap","volume":"24","author":"Demner-Fushman","year":"2017","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B11","first-page":"195","article-title":"\u201cThe KnowledgeMap project: development of a concept-based medical school curriculum database,\u201d","author":"Denny","year":"2003","journal-title":"AMIA Annu. Symp. Proc. AMIA Symp"},{"key":"B12","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1007\/978-1-4939-0847-9_16","article-title":"Natural language processing in biomedicine: a unified system architecture overview","volume":"1168","author":"Doan","year":"2014","journal-title":"Methods Mol. Biol"},{"key":"B13","doi-asserted-by":"publisher","first-page":"48","DOI":"10.15265\/IYS-2016-s006","article-title":"Electronic health records: then, now, and in the future","volume":"25","author":"Evans","year":"2016","journal-title":"Yearb. Med. Inform"},{"key":"B14","first-page":"438","article-title":"Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python","volume":"2021","author":"Eyre","year":"2021","journal-title":"AMIA Annu. Symp. Proc"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1219","DOI":"10.1093\/annonc\/mds603","article-title":"Docetaxel-cisplatin might be superior to docetaxel-capecitabine in the first-line treatment of metastatic triple-negative breast cancer","volume":"24","author":"Fan","year":"2013","journal-title":"Ann. Oncol."},{"key":"B16","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1093\/jamia\/ocv180","article-title":"Extracting information from the text of electronic medical records to improve case detection: a systematic review","volume":"23","author":"Ford","year":"2016","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B17","first-page":"270","article-title":"\u201cA broad-coverage natural language processing system,\u201d","author":"Friedman","year":"2000","journal-title":"Proc. AMIA Symp"},{"key":"B18","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1136\/jamia.1994.95236146","article-title":"A general natural-language text processor for clinical radiology","volume":"1","author":"Friedman","year":"1994","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B19","doi-asserted-by":"publisher","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","article-title":"Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives","volume":"13","author":"Gehrmann","year":"2018","journal-title":"PLoS ONE"},{"key":"B20","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1200\/CCI.19.00134","article-title":"Electronic medical record search engine (EMERSE): an information retrieval tool for supporting cancer research","volume":"4","author":"Hanauer","year":"2020","journal-title":"JCO Clin. Cancer Inform"},{"key":"B21","first-page":"15","article-title":"\u201cEvaluation of medical concept annotation systems on clinical records,\u201d","author":"Hassanzadeh","year":"2016","journal-title":"Proceedings of the Australasian Language Technology Association Workshop 2016, Melbourne, Australia"},{"key":"B22","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1093\/jamia\/ocz119","article-title":"Real world evidence in cardiovascular medicine: ensuring data validity in electronic health record-based studies","volume":"26","author":"Hernandez-Boussard","year":"2019","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B23","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1097\/MLR.0b013e31829b1dbd","article-title":"Caveats for the use of operational electronic health record data in comparative effectiveness research","volume":"51","author":"Hersh","year":"2013","journal-title":"Med. Care"},{"key":"B24","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci. Data"},{"key":"B25","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1093\/jamia\/ocx084","article-title":"The MIMIC Code Repository: enabling reproducibility in critical care research","volume":"25","author":"Johnson","year":"2018","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B26","first-page":"56","article-title":"The open biomedical annotator","volume":"2009","author":"Jonquet","year":"2009","journal-title":"Summit Transl. Bioinforma"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.5544","article-title":"Natural language processing-enabled and conventional data capture methods for input to electronic health records: a comparative usability study","volume":"4","author":"Kaufman","year":"2016","journal-title":"JMIR Med. Inform"},{"key":"B28","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1136\/amiajnl-2013-001625","article-title":"Combining rules and machine learning for extraction of temporal expressions and events from clinical narratives","volume":"20","author":"Kova\u010devi\u0107","year":"2013","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B29","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review","volume":"73","author":"Kreimeyer","year":"2017","journal-title":"J. Biomed. Inform"},{"key":"B30","doi-asserted-by":"publisher","first-page":"57365","DOI":"10.18632\/oncotarget.17071","article-title":"Docetaxel versus docetaxel plus cisplatin for non-small-cell lung cancer: a meta-analysis of randomized clinical trials","volume":"8","author":"Li","year":"2017","journal-title":"Oncotarget"},{"key":"B31","unstructured":"LiI.\n            YouK.\n            TangX.\n            QiaoY.\n            HuangL.\n            HsiehC. C.\n          EHRKit: A Python Natural Language Processing Toolkit for Electronic Health Record Texts. arXiv2022"},{"key":"B32","doi-asserted-by":"publisher","first-page":"W566","DOI":"10.1093\/nar\/gkz386","article-title":"Doc2Hpo: a web application for efficient and accurate HPO concept curation","volume":"47","author":"Liu","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"B33","first-page":"149","article-title":"An information extraction framework for cohort identification using electronic health records","volume":"2013","author":"Liu","year":"2013","journal-title":"AMIA Jt. Summits Transl. Sci. Proc."},{"key":"B34","doi-asserted-by":"publisher","first-page":"92","DOI":"10.18653\/v1\/W15-3810","article-title":"Exploiting task-oriented resources to learn word embeddings for clinical abbreviation expansion","volume":"15","author":"Liu","year":"2015","journal-title":"Proc. BioNLP"},{"key":"B35","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1016\/j.jbi.2018.06.007","article-title":"A novel framework for biomedical entity sense induction","volume":"84","author":"Lossio-Ventura","year":"2018","journal-title":"J. Biomed. Inform"},{"key":"B36","doi-asserted-by":"crossref","first-page":"1548","DOI":"10.1109\/BIBM47256.2019.8983406","article-title":"\u201cClinical named-entity recognition: a short comparison,\u201d","author":"Lossio-Ventura","year":"2019","journal-title":"2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), San Diego, CA, USA"},{"key":"B37","unstructured":"A Tool For Recognizing UMLS Concepts in Text2021"},{"key":"B38","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1055\/s-0038-1638592","article-title":"Extracting information from textual documents in the electronic health record: a review of recent research","volume":"17","author":"Meystre","year":"2008","journal-title":"Yearb. Med. Inform"},{"key":"B39","first-page":"1744","article-title":"\u201cUmlsBERT: clinical domain knowledge augmentation of contextual embeddings using the unified medical language system metathesaurus,\u201d","author":"Michalopoulos","year":"2021","journal-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 2021"},{"key":"B40","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1001\/jama.2013.393","article-title":"The inevitable application of big data to health care","volume":"309","author":"Murdoch","year":"2013","journal-title":"JAMA"},{"key":"B41","unstructured":"2021"},{"key":"B42","doi-asserted-by":"crossref","first-page":"319","DOI":"10.18653\/v1\/W19-5034","article-title":"\u201cScispaCy: fast and robust models for biomedical natural language processing,\u201d","author":"Neumann","year":"2019","journal-title":"Proc. 18th BioNLP Workshop Shar. Task"},{"key":"B43","unstructured":"\u201cSimple and efficient algorithm for approximate dictionary matching,\u201d\n            OkazakiN.\n            TsujiiJ.\n          16310348Proceedings of the 23rd International Conference on Computational Linguistics, Beijing, China2010"},{"key":"B44","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3115\/v1\/S14-2007","article-title":"\u201cSemEval-2014 task 7: analysis of clinical text,\u201d","author":"Pradhan","year":"2014","journal-title":"Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland"},{"key":"B45","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1136\/amiajnl-2013-002544","article-title":"Evaluating the state of the art in disorder recognition and normalization of the clinical narrative","volume":"22","author":"Pradhan","year":"2015","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B46","unstructured":"System for Medical Concept Extraction and Linking2021"},{"key":"B47","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1186\/s12911-018-0654-2","article-title":"Comparison of MetaMap and cTAKES for entity extraction in clinical notes","volume":"18","author":"Re\u00e1tegui","year":"2018","journal-title":"BMC Med. Inform. Decis. Mak"},{"key":"B48","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B49","unstructured":"SpaCy Models for Biomedical Text Processing2021"},{"key":"B50","article-title":"\u201cQuickUMLS : a fast, unsupervised approach for medical concept extraction,\u201d","volume":"4","author":"Soldaini","year":"2016","journal-title":"Medical Information Retrieval (MedIR) Workshop, Pisa, Italy"},{"key":"B51","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1093\/jamia\/ocx132","article-title":"CLAMP - a toolkit for efficiently building customized clinical natural language processing pipelines","volume":"25","author":"Soysal","year":"2017","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B52","unstructured":"Natural Language Processing in Python2021"},{"key":"B53","doi-asserted-by":"publisher","first-page":"1962","DOI":"10.1093\/bioinformatics\/bty009","article-title":"Enhanced functionalities for annotating and indexing clinical text with the NCBO Annotator","volume":"34","author":"Tchechmedjiev","year":"2018","journal-title":"Bioinforma. Oxf. Engl"},{"key":"B54","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1186\/s12859-015-0871-y","article-title":"NOBLE - Flexible concept recognition for large-scale biomedical natural language processing","volume":"17","author":"Tseytlin","year":"2016","journal-title":"BMC Bioinformatics"},{"key":"B55","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1136\/jamia.2010.003947","article-title":"Extracting medication information from clinical text","volume":"17","author":"Uzuner","year":"2010","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B56","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b.2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B57","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1007\/s13042-015-0426-6","article-title":"A comparative study for biomedical named entity recognition","volume":"9","author":"Wang","year":"2018","journal-title":"Int. J. Mach. Learn. Cybern"},{"key":"B58","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J. Biomed. Inform"},{"key":"B59","first-page":"997","article-title":"A comparative study of current Clinical Natural Language Processing systems on handling abbreviations in discharge summaries","volume":"2012","author":"Wu","year":"2012","journal-title":"AMIA Annu. Symp. Proc. AMIA Symp"},{"key":"B60","doi-asserted-by":"publisher","first-page":"797","DOI":"10.1001\/jamaoncol.2016.0213","article-title":"Natural language processing in oncology: a review","volume":"2","author":"Yim","year":"2016","journal-title":"JAMA Oncol"},{"key":"B61","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6947-6-30","article-title":"Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system","volume":"6","author":"Zeng","year":"2006","journal-title":"BMC Med. Inform. Decis. Mak"},{"key":"B62","doi-asserted-by":"publisher","first-page":"1892","DOI":"10.1093\/jamia\/ocab090","article-title":"Biomedical and clinical English model packages for the Stanza Python NLP library","volume":"28","author":"Zhang","year":"2021","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B63","doi-asserted-by":"publisher","first-page":"1208","DOI":"10.1093\/jamia\/ocac040","article-title":"CancerBERT: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records","volume":"29","author":"Zhou","year":"2022","journal-title":"J. Am. Med. Inform. Assoc"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.1051724\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,13]],"date-time":"2023-01-13T05:23:14Z","timestamp":1673587394000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.1051724\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,13]]},"references-count":63,"alternative-id":["10.3389\/frai.2022.1051724"],"URL":"https:\/\/doi.org\/10.3389\/frai.2022.1051724","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,13]]},"article-number":"1051724"}}