{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T09:41:32Z","timestamp":1782985292080,"version":"3.54.5"},"reference-count":43,"publisher":"Oxford University Press (OUP)","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The 2010 i2b2\/VA Workshop on Natural Language Processing Challenges for Clinical Records presented three tasks: a concept extraction task focused on the extraction of medical concepts from patient reports; an assertion classification task focused on assigning assertion types for medical problem concepts; and a relation classification task focused on assigning relation types that hold between medical problems, tests, and treatments. i2b2 and the VA provided an annotated reference standard corpus for the three tasks. Using this reference standard, 22 systems were developed for concept extraction, 21 for assertion classification, and 16 for relation classification.<\/jats:p>\n               <jats:p>These systems showed that machine learning approaches could be augmented with rule-based systems to determine concepts, assertions, and relations. Depending on the task, the rule-based systems can either provide input for machine learning or post-process the output of machine learning. Ensembles of classifiers, information from unlabeled data, and external knowledge sources can help when the training data are inadequate.<\/jats:p>","DOI":"10.1136\/amiajnl-2011-000203","type":"journal-article","created":{"date-parts":[[2011,6,18]],"date-time":"2011-06-18T02:08:15Z","timestamp":1308362895000},"page":"552-556","source":"Crossref","is-referenced-by-count":753,"title":["2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text"],"prefix":"10.1093","volume":"18","author":[{"given":"\u00d6zlem","family":"Uzuner","sequence":"first","affiliation":[{"name":"Department of Information Studies, University at Albany, State University of New York, Albany, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brett R","family":"South","sequence":"first","affiliation":[{"name":"VA Salt Lake City Health Care System, Salt Lake City, Utah, USA"},{"name":"Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA"},{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuying","family":"Shen","sequence":"first","affiliation":[{"name":"VA Salt Lake City Health Care System, Salt Lake City, Utah, USA"},{"name":"Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA"},{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott L","family":"DuVall","sequence":"first","affiliation":[{"name":"VA Salt Lake City Health Care System, Salt Lake City, Utah, USA"},{"name":"Department of Internal Medicine, University of Utah, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2011,6,16]]},"reference":[{"key":"2020040605151544300_b1","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1136\/jamia.2010.003947","article-title":"Extracting medication information from clinical text","volume":"17","author":"Uzuner","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b2","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1197\/jamia.M2444","article-title":"Evaluating the state-of-the-art in automatic de-identification","volume":"14","author":"Uzuner","year":"2007","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b3","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1197\/jamia.M2408","article-title":"Identifying patient smoking status from medical discharge summaries","volume":"15","author":"Uzuner","year":"2008","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b4","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1197\/jamia.M3115","article-title":"Recognizing obesity and co-morbidities in sparse data","volume":"16","author":"Uzuner","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b5","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1136\/jamia.2010.004200","article-title":"Community annotation experiment for ground truth generation for the i2b2 medication challenge","volume":"17","author":"Uzuner","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b6","first-page":"97","article-title":"A shared task involving multi-label classification of clinical free text","volume-title":"Proceedings of ACL","author":"Pestian","year":"2007"},{"issue":"6 Suppl 1","key":"2020040605151544300_b7","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1471-2105-6-S1-S1","article-title":"Overview of BioCreAtIvE: critical assessment of information extraction for biology","author":"Hirschman","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2020040605151544300_b8","first-page":"773","article-title":"Enhancing access to the bibliome: the TREC genomics track","volume":"11","author":"Hersh","year":"2004","journal-title":"Stud Health Technol Inform"},{"key":"2020040605151544300_b9","doi-asserted-by":"crossref","first-page":"466","DOI":"10.3115\/992628.992709","article-title":"Message understanding conference-6: a brief history","volume-title":"16th Conference on Computational Linguistics (COLING)","author":"Grishman","year":"1996"},{"key":"2020040605151544300_b10","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/0306-4573(94)00048-8","article-title":"Reflections on TREC","volume":"31","author":"Sparck Jones","year":"1995","journal-title":"Inform Process Manag"},{"key":"2020040605151544300_b11","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","year":"2001","journal-title":"J Biomed Inform"},{"key":"2020040605151544300_b12","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1197\/jamia.M2950","article-title":"Machine Learning and Rule-Based Approaches to Assertion Classification","volume":"16","author":"Uzuner","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b13","first-page":"81","volume-title":"ConText: An Algorithm for Identifying Contextual Features from Clinical Text","author":"Chapman","year":"2007"},{"key":"2020040605151544300_b14","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.artmed.2010.05.006","article-title":"Semantic Relations for Problem-Oriented Medical Records","volume":"50","author":"Uzuner","year":"2010","journal-title":"Artif Intell Med"},{"key":"2020040605151544300_b15","volume-title":"Statistical Applications for Health Information Management","author":"Osborn","year":"2005","edition":"2nd edn"},{"key":"2020040605151544300_b16","article-title":"Hybrid approaches to concept extraction and assertion classification - vanderbilt's systems for 2010 I2B2 NLP Challenge","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Jiang","year":"2010"},{"key":"2020040605151544300_b17","article-title":"Erasmus MC approaches to the i2b2 Challenge","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Kang","year":"2010"},{"key":"2020040605151544300_b18","article-title":"Concept identification and assertion classification in patient health records","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Gurulingappa","year":"2010"},{"key":"2020040605151544300_b19","article-title":"I2b2 challenges in clinical natural language processing 2010","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Patrick","year":"2010"},{"key":"2020040605151544300_b20","article-title":"BioTagger-GM for detecting clinical concepts in electronic medical reports","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Torii","year":"2010"},{"key":"2020040605151544300_b21","article-title":"Can distributional statistics aid clinical concept extraction?","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Jonnalagadda","year":"2010"},{"key":"2020040605151544300_b22","article-title":"TTI's systems for 2010 i2b2\/VA challenge","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Sasaki","year":"2010"},{"key":"2020040605151544300_b23","article-title":"Extraction of medical concepts, assertions, and relations from discharge summaries for the fourth i2b2\/VA shared task","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Roberts","year":"2010"},{"key":"2020040605151544300_b24","article-title":"The emory system for extracting medical concepts at 2010 i2b2 challenge: integrating natural language processing and machine learning techniques","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Pai","year":"2010"},{"key":"2020040605151544300_b25","article-title":"NRC at i2b2: one challenge, three practical tasks, nine statistical systems, hundreds of clinical records, millions of useful features","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"deBruijn","year":"2010"},{"key":"2020040605151544300_b26","doi-asserted-by":"crossref","first-page":"S14","DOI":"10.1186\/1471-2105-6-S1-S14","article-title":"ProMiner: rule-based protein and gene entity recognition","volume":"6","author":"Hanisch","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2020040605151544300_b27","first-page":"104","article-title":"Biomedical named entity recognition using conditional random fields and rich feature sets","volume-title":"Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA)","author":"Settles","year":"2004"},{"key":"2020040605151544300_b28","unstructured":"LingPipe. http:\/\/alias-i.com\/lingpipe (accessed 29 Nov 2010)."},{"key":"2020040605151544300_b29","first-page":"65","article-title":"Automatically adapting an NLP core engine to the biology domain","volume-title":"Proceedings of the Joint BioLINK\/Bio-Ontologies Meeting 2006","author":"Buyko","year":"2006"},{"key":"2020040605151544300_b30","first-page":"131","article-title":"Peregrine: lightweight gene name normalization by dictionary lookup","volume-title":"Proceedings of the BioCreAtIvE II Workshop","author":"Schuemie","year":"2007"},{"key":"2020040605151544300_b31","unstructured":"StanfordNer. http:\/\/nlp.stanford.edu\/software\/CRF-NER.shtml (accessed 29 Nov 2010)."},{"key":"2020040605151544300_b32","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1197\/jamia.M1176","article-title":"\u201cUnderstanding\u201d medical school curriculum content using knowledgeMap","volume":"10","author":"Denny","year":"2003","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b33","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1136\/jamia.1994.95236146","article-title":"A general natural language text processor for clinical radiology","volume":"1","author":"Friedman","year":"1994","journal-title":"J Am Med Inform Assoc"},{"key":"2020040605151544300_b34","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-12116-6_19","article-title":"A distributional semantics approach to simultaneous recognition of multiple classes of named entities","volume-title":"Computational Linguistics and Intelligent Text Processing (CICLing)","author":"Jonnalagadda","year":"2010"},{"key":"2020040605151544300_b35","article-title":"NLM's system description for the fourth i2b2\/VA challenge","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Demner-Fushman","year":"2010"},{"key":"2020040605151544300_b36","article-title":"CARAMBA: concept, assertion, and relation annotation using machine-learning based approaches","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Grouin","year":"2010"},{"key":"2020040605151544300_b37","article-title":"Salt lake city VA's challenge submissions","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Divita","year":"2010"},{"key":"2020040605151544300_b38","article-title":"OHSU\/portland VAMC team participation in the 2010 i2b2\/VA challenge tasks","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Cohen","year":"2010"},{"key":"2020040605151544300_b39","article-title":"I2B2 2010 challenge: machine learning for information extraction from patient records","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Anick","year":"2010"},{"key":"2020040605151544300_b40","article-title":"A hybrid approach to extract structured information from narrative clinical discharge summaries","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Chang","year":"2010"},{"key":"2020040605151544300_b41","article-title":"Determining assertion status for medical problems in clinical records","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Clark","year":"2010"},{"key":"2020040605151544300_b42","article-title":"Concept, assertion and relation extraction at the 2010 i2b2 relation extraction challenge using parsing information and dictionaries","volume-title":"Proceedings of the 2010 i2b2\/VA Workshop on Challenges in Natural Language Processing for Clinical Data","author":"Solt","year":"2010"},{"key":"2020040605151544300_b43","first-page":"17","article-title":"Effective mapping of biomedical text to the UMLS metathesaurus: the metamap program","volume":"2001","author":"Aronson","year":"2001","journal-title":"AMIA Annu Symp Proc"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/18\/5\/552\/33015279\/18-5-552.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/18\/5\/552\/33015279\/18-5-552.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,4,6]],"date-time":"2020-04-06T09:15:36Z","timestamp":1586164536000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/18\/5\/552\/830538"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,6,16]]},"references-count":43,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2011,6,16]]},"published-print":{"date-parts":[[2011,9,1]]}},"URL":"https:\/\/doi.org\/10.1136\/amiajnl-2011-000203","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2011,9]]},"published":{"date-parts":[[2011,6,16]]}}}