{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T15:12:05Z","timestamp":1772032325723,"version":"3.50.1"},"reference-count":62,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2016,10,4]],"date-time":"2016-10-04T00:00:00Z","timestamp":1475539200000},"content-version":"vor","delay-in-days":775,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2015,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Objective The ShARe\/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification\/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners.<\/jats:p>\n               <jats:p>Materials and methods We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text\u2014199 clinical notes for training and 99 for testing (roughly 180\u2005K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance.<\/jats:p>\n               <jats:p>Results For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59.<\/jats:p>\n               <jats:p>Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources.<\/jats:p>\n               <jats:p>Conclusions The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.<\/jats:p>","DOI":"10.1136\/amiajnl-2013-002544","type":"journal-article","created":{"date-parts":[[2014,8,22]],"date-time":"2014-08-22T07:27:08Z","timestamp":1408692428000},"page":"143-154","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":78,"title":["Evaluating the state of the art in disorder recognition and normalization of the clinical narrative"],"prefix":"10.1093","volume":"22","author":[{"given":"Sameer","family":"Pradhan","sequence":"first","affiliation":[{"name":"Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA"}]},{"given":"No\u00e9mie","family":"Elhadad","sequence":"additional","affiliation":[{"name":"Columbia University, New York, New York, USA"}]},{"given":"Brett R","family":"South","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah, USA"}]},{"given":"David","family":"Martinez","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Australia"}]},{"given":"Lee","family":"Christensen","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah, USA"}]},{"given":"Amy","family":"Vogel","sequence":"additional","affiliation":[{"name":"Columbia University, New York, New York, USA"}]},{"given":"Hanna","family":"Suominen","sequence":"additional","affiliation":[{"name":"NICTA, The Australian National University, and University of Canberra, Canberra, Australian Capital Territory, Australia"}]},{"given":"Wendy W","family":"Chapman","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, Utah, USA"}]},{"given":"Guergana","family":"Savova","sequence":"additional","affiliation":[{"name":"Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA"}]}],"member":"286","published-online":{"date-parts":[[2014,8,21]]},"reference":[{"key":"2020110613001269700_R1","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1016\/j.jbi.2009.08.007","article-title":"What can natural language processing do for clinical decision support?","volume":"42","author":"Demner-Fushman","year":"2009","journal-title":"J Biomed Inform"},{"key":"2020110613001269700_R2","first-page":"139","article-title":"Teaching and learning through clinical report-writing genres","volume":"16","author":"Oglensky","year":"2009","journal-title":"Int J Learn"},{"key":"2020110613001269700_R3","volume-title":"Clinical ethics and the necessity of stories","author":"Zaner","year":"2011"},{"key":"2020110613001269700_R4","doi-asserted-by":"crossref","first-page":"922","DOI":"10.1136\/amiajnl-2012-001317","article-title":"Towards comprehensive syntactic and semantic annotations of the clinical narrative","volume":"20","author":"Albright","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R5","article-title":"Discovering temporal narrative containers in clinical text","author":"Miller"},{"key":"2020110613001269700_R6","author":"THYME \u2013 Temporal Histories of Your Medical Event"},{"key":"2020110613001269700_R7","first-page":"143","article-title":"Temporal annotation in the clinical domain","volume":"2","author":"Styler","year":"2014","journal-title":"Trans Comput Linguist"},{"key":"2020110613001269700_R8","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R9","author":"i2b2 \u2013 Informatics for Integrating Biology & the Bedside"},{"key":"2020110613001269700_R10","author":"Elhadad"},{"key":"2020110613001269700_R11","author":"SHARPn: Strategic Health IT Advanced Research Projects"},{"key":"2020110613001269700_R12","doi-asserted-by":"crossref","first-page":"e341","DOI":"10.1136\/amiajnl-2013-001939","article-title":"Normalization and standardization of electronic health records for high-throughput phenotyping: the SHARPn consortium","volume":"20","author":"Pathak","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R13","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1136\/amiajnl-2011-000465","article-title":"Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions","volume":"18","author":"Chapman","year":"2011","journal-title":"J Am Med Informatics Assoc"},{"key":"2020110613001269700_R14","first-page":"497","article-title":"A highly specific algorithm for identifying asthma cases and controls for genome-wide association studies","volume":"2009","author":"Pacheco","year":"2009","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110613001269700_R15","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1186\/1471-2415-11-32","article-title":"Cataract research using electronic health records","volume":"11","author":"Waudby","year":"2011","journal-title":"BMC Ophthalmol"},{"key":"2020110613001269700_R16","doi-asserted-by":"crossref","first-page":"79re1","DOI":"10.1126\/scitranslmed.3001807","article-title":"Electronic medical records for genetic research: results of the eMERGE consortium","volume":"3","author":"Kho","year":"2011","journal-title":"Sci Transl Med"},{"key":"2020110613001269700_R17","doi-asserted-by":"crossref","first-page":"568","DOI":"10.1136\/jamia.2010.004366","article-title":"Leveraging informatics for genetic studies: use of the electronic medical record to enable a genome-wide association study of peripheral arterial disease","volume":"17","author":"Kullo","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R18","doi-asserted-by":"crossref","first-page":"e69932","DOI":"10.1371\/journal.pone.0069932","article-title":"Automatic prediction of rheumatoid arthritis disease activity from the electronic medical records","volume":"8","author":"Lin","year":"2013","journal-title":"PLoS ONE"},{"key":"2020110613001269700_R19","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1136\/amiajnl-2011-000208","article-title":"Facilitating pharmacogenetic studies using electronic health records and natural-language processing: a case study of warfarin","volume":"18","author":"Xu","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R20","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1038\/clpt.2010.260","article-title":"The emerging role of electronic medical records in pharmacogenomics","volume":"89","author":"Wilke","year":"2011","journal-title":"Clin Pharmacol Ther"},{"key":"2020110613001269700_R21","author":"CoNLL \u2013 Computational Natural Language Learning"},{"key":"2020110613001269700_R22","author":"SemEval \u2013 Semantic Evaluations"},{"key":"2020110613001269700_R23","author":"BioNLP"},{"key":"2020110613001269700_R24","author":"BioCreAtIvE"},{"key":"2020110613001269700_R25","author":"i2b2 Shared Tasks"},{"key":"2020110613001269700_R26","author":"SNOMED Clinical Terms (SNOMED CT)"},{"key":"2020110613001269700_R27","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1016\/j.jbi.2003.11.002","article-title":"Exploring semantic groups through visual approaches","volume":"36","author":"Bodenreider","year":"2003","journal-title":"J Biomed Inform"},{"key":"2020110613001269700_R28","author":"UMLS Metathesaurus"},{"key":"2020110613001269700_R29","first-page":"1","article-title":"Task 1: ShARe\/CLEF eHealth Evaluation Lab 2013","author":"Pradhan","year":"2013"},{"key":"2020110613001269700_R30","author":"MeSH \u2013 Medical Subject Headings"},{"key":"2020110613001269700_R31","author":"RxNorm"},{"key":"2020110613001269700_R32","first-page":"270","article-title":"A broad-coverage natural language processing system","author":"Friedman","year":"2000","journal-title":"Proc AMIA Symp"},{"key":"2020110613001269700_R33","first-page":"17","article-title":"Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program","author":"Aronson","year":"2001","journal-title":"Proc AMIA Symp"},{"key":"2020110613001269700_R34","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R35","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R36","doi-asserted-by":"crossref","first-page":"882","DOI":"10.1136\/amiajnl-2012-001350","article-title":"Knowledge-based biomedical word sense disambiguation: an evaluation and application to clinical document classification","volume":"20","author":"Garla","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001269700_R37","doi-asserted-by":"crossref","first-page":"2909","DOI":"10.1093\/bioinformatics\/btt474","article-title":"DNorm: disease name normalization with pairwise learning to rank","volume":"29","author":"Leaman","year":"2013","journal-title":"Bioinformatics"},{"key":"2020110613001269700_R38","first-page":"91","article-title":"An improved corpus of disease mentions in PubMed citations","author":"Do\u011fan","year":"2012"},{"key":"2020110613001269700_R39","first-page":"82","article-title":"Enabling recognition of diseases in biomedical text with machine learning: corpus and benchmark","author":"Leaman","year":"2009"},{"key":"2020110613001269700_R40","first-page":"15","article-title":"An empirical evaluation of resources for the identification of diseases and adverse effects in biomedical literature","author":"Gurulingappa","year":"2010"},{"key":"2020110613001269700_R41","author":"Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC-II)."},{"key":"2020110613001269700_R42","first-page":"1","article-title":"Overview of the ShARe\/CLEF EHealth Evaluation Lab 2013","author":"Suominen","year":"2013"},{"key":"2020110613001269700_R43","first-page":"947","article-title":"More accurate tests for the statistical significance of result differences","author":"Yeh","year":"2000"},{"key":"2020110613001269700_R44","article-title":"Recognizing and encoding disorder concepts in clinical text using machine learning and vector space","author":"Tang","year":"2013"},{"key":"2020110613001269700_R45","first-page":"467","article-title":"Class-based n-gram models of natural language","volume":"18","author":"Brown","year":"1992","journal-title":"Comput Linguist"},{"key":"2020110613001269700_R46","first-page":"143","article-title":"Towards robust linguistic analysis using OntoNotes","author":"Pradhan","year":"2013"},{"key":"2020110613001269700_R47","first-page":"2","article-title":"Combining MetaMap and cTAKES in disorder recognition: THCIB at CLEF eHealth Lab 2013 Task 1","author":"Xia","year":"2013"},{"key":"2020110613001269700_R48","article-title":"Disorder concept identification from clinical notes an experience with the ShARe\/CLEF 2013 challenge","author":"Fan","year":"2013"},{"key":"2020110613001269700_R49","first-page":"1","article-title":"Performance of a multi-class biomedical tagger on clinical records","author":"Ramanan","year":"2013"},{"key":"2020110613001269700_R50","article-title":"ShARe\/CLEF Task 1 Working Notes Team UCSC introduction to Task 1","author":"Wang","year":"2013"},{"key":"2020110613001269700_R51","article-title":"Disorder normalization in clinical notes with DNorm","author":"Leaman","year":"2013"},{"key":"2020110613001269700_R52","article-title":"ShARe\/CLEF eHealth 2013 named entity recognition and normalization of disorders challenge","author":"Patrick","year":"2013"},{"key":"2020110613001269700_R53","article-title":"Using relations for identification and normalization of disorders: team CLEAR in the ShARe\/CLEF 2013 eHealth Evaluation Lab","author":"Gung","year":"2013"},{"key":"2020110613001269700_R54","author":"The ClearNLP Project"},{"key":"2020110613001269700_R55","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1613\/jair.105","article-title":"Solving multiclass learning problems via error-correcting output codes","volume":"2","author":"Dietterich","year":"1995","journal-title":"J Artif Intell Res"},{"key":"2020110613001269700_R56","unstructured":"Loper\n              E\n            \n          . Encoding structured output values [Ph.D. Thesis]. University of Pennsylvania. 2008."},{"key":"2020110613001269700_R57","article-title":"Identify disorders in health records using conditional random fields and metamap","author":"Zuccon","year":"2013"},{"key":"2020110613001269700_R58","article-title":"Integrated cTAKES for concept mention detection and normalization","author":"Liu","year":"2013"},{"key":"2020110613001269700_R59","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1145\/318723.318728","article-title":"Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone","author":"Lesk","year":"1986"},{"key":"2020110613001269700_R60","article-title":"Evaluation of YTEX and MetaMap for clinical concept recognition","author":"Osborne","year":"2013"},{"key":"2020110613001269700_R61","first-page":"1","article-title":"FACTORIE: probabilistic programming via imperatively defined factor graphs","author":"McCallum","year":"2009"},{"key":"2020110613001269700_R62","author":"FACTORIE Toolkit"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/1\/143\/34145292\/amiajnl-2013-002544.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/1\/143\/34145292\/amiajnl-2013-002544.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:42:35Z","timestamp":1604688155000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/22\/1\/143\/832969"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,8,21]]},"references-count":62,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2014,8,21]]},"published-print":{"date-parts":[[2015,1,1]]}},"URL":"https:\/\/doi.org\/10.1136\/amiajnl-2013-002544","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,1]]},"published":{"date-parts":[[2014,8,21]]}}}