{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T15:49:45Z","timestamp":1773071385962,"version":"3.50.1"},"reference-count":58,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2016,10,4]],"date-time":"2016-10-04T00:00:00Z","timestamp":1475539200000},"content-version":"vor","delay-in-days":714,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2015,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Objective The trade-off between the speed and simplicity of dictionary-based term recognition and the richer linguistic information provided by more advanced natural language processing (NLP) is an area of active discussion in clinical informatics. In this paper, we quantify this trade-off among text processing systems that make different trade-offs between speed and linguistic understanding. We tested both types of systems in three clinical research tasks: phase IV safety profiling of a drug, learning adverse drug\u2013drug interactions, and learning used-to-treat relationships between drugs and indications.<\/jats:p><jats:p>Materials We first benchmarked the accuracy of the NCBO Annotator and REVEAL in a manually annotated, publically available dataset from the 2008 i2b2 Obesity Challenge. We then applied the NCBO Annotator and REVEAL to 9 million clinical notes from the Stanford Translational Research Integrated Database Environment (STRIDE) and used the resulting data for three research tasks.<\/jats:p><jats:p>Results There is no significant difference between using the NCBO Annotator and REVEAL in the results of the three research tasks when using large datasets. In one subtask, REVEAL achieved higher sensitivity with smaller datasets.<\/jats:p><jats:p>Conclusions For a variety of tasks, employing simple term recognition methods instead of advanced NLP methods results in little or no impact on accuracy when using large datasets. Simpler dictionary-based methods have the advantage of scaling well to very large datasets. Promoting the use of simple, dictionary-based methods for population level analyses can advance adoption of NLP in practice.<\/jats:p>","DOI":"10.1136\/amiajnl-2014-002902","type":"journal-article","created":{"date-parts":[[2014,10,22]],"date-time":"2014-10-22T03:14:01Z","timestamp":1413947641000},"page":"121-131","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":30,"title":["Functional evaluation of out-of-the-box text-mining tools for data-mining tasks"],"prefix":"10.1093","volume":"22","author":[{"given":"Kenneth","family":"Jung","sequence":"first","affiliation":[{"name":"Program in Biomedical Informatics, Stanford University, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paea","family":"LePendu","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Srinivasan","family":"Iyer","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anna","family":"Bauer-Mehren","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bethany","family":"Percha","sequence":"additional","affiliation":[{"name":"Program in Biomedical Informatics, Stanford University, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nigam H","family":"Shah","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2014,10,21]]},"reference":[{"key":"2020110613001288500_R1","first-page":"100","article-title":"Mining electronic health records for adverse drug effects using regression based methods","author":"Harpaz","year":"2010"},{"key":"2020110613001288500_R2","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1038\/clpt.2012.54","article-title":"Detection of pharmacovigilance-related adverse events using electronic health records and automated methods","volume":"92","author":"Haerian","year":"2012","journal-title":"Clin Pharmacol Ther"},{"key":"2020110613001288500_R3","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1197\/jamia.M3028","article-title":"Active computerized pharmacovigilance using natural language processing, statistics, and electronic health records: a feasibility study","volume":"16","author":"Wang","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R4","first-page":"1","author":"Friedman","year":"2009"},{"key":"2020110613001288500_R5","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1136\/amiajnl-2012-001119","article-title":"Comparative analysis of pharmacovigilance methods in the detection of adverse drug reactions using electronic medical records","volume":"20","author":"Liu","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"(Suppl 1)","key":"2020110613001288500_R6","first-page":"1","article-title":"The U.S. Food and Drug Administration's Mini-Sentinel program: status and direction","volume":"21","author":"Platt","year":"2012","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"2020110613001288500_R7","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1136\/amiajnl-2013-001612","article-title":"Mining clinical text for signals of adverse drug-drug interactions","volume":"21","author":"Iyer","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R8","doi-asserted-by":"crossref","first-page":"e1002614","DOI":"10.1371\/journal.pcbi.1002614","article-title":"Literature based drug interaction prediction with clinical assessment using electronic medical records: novel myopathy associated drug interactions","volume":"8","author":"Duke","year":"2012","journal-title":"PLoS Comput Biol"},{"issue":"(e2)","key":"2020110613001288500_R9","doi-asserted-by":"crossref","first-page":"e297","DOI":"10.1136\/amiajnl-2013-001933","article-title":"Identifying phenotypic signatures of neuropsychiatric disorders from electronic medical records","volume":"20","author":"Lyalina","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"(e2)","key":"2020110613001288500_R10","doi-asserted-by":"crossref","first-page":"e341","DOI":"10.1136\/amiajnl-2013-001939","article-title":"Normalization and standardization of electronic health records for high-throughput phenotyping: the SHARPn consortium","volume":"20","author":"Pathak","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"(e2)","key":"2020110613001288500_R11","doi-asserted-by":"crossref","first-page":"e334","DOI":"10.1136\/amiajnl-2013-001999","article-title":"Automated extraction of clinical traits of multiple sclerosis in electronic medical records","volume":"20","author":"Davis","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R12","doi-asserted-by":"crossref","first-page":"e89324","DOI":"10.1371\/journal.pone.0089324","article-title":"Automated detection of off-label drug use","volume":"9","author":"Jung","year":"2014","journal-title":"PloS ONE"},{"key":"2020110613001288500_R13","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1197\/jamia.M2401","article-title":"Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study","volume":"15","author":"Chen","year":"2008","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R14","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/j.jbi.2012.11.006","article-title":"Detecting concept relations in clinical text: insights from a state-of-the-art model","volume":"46","author":"Zhu","year":"2013","journal-title":"J Biomed Inform"},{"key":"2020110613001288500_R15","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1136\/amiajnl-2011-000150","article-title":"Machine-learned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010","volume":"18","author":"de Bruijn","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R16","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1136\/jamia.2010.003939","article-title":"High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge","volume":"17","author":"Patrick","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R17","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/1472-6947-10-10","article-title":"Assessing the accuracy of an inter-institutional automated patient-specific health problem list","volume":"10","author":"Poissant","year":"2010","journal-title":"BMC Med Inform Decis Mak"},{"key":"2020110613001288500_R18","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1097\/01.mlr.0000160417.39497.a9","article-title":"Accuracy of ICD-9-CM codes for identifying cardiovascular and stroke risk factors","volume":"43","author":"Birman-Deych","year":"2005","journal-title":"Med Care"},{"issue":"(e1)","key":"2020110613001288500_R19","doi-asserted-by":"crossref","first-page":"e162","DOI":"10.1136\/amiajnl-2011-000583","article-title":"Portability of an algorithm to identify rheumatoid arthritis in electronic health records","volume":"19","author":"Carroll","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R20","first-page":"1564","article-title":"Extracting and integrating data from entire electronic health records for detecting colorectal cancer cases","volume":"2011","author":"Xu","year":"2011","journal-title":"AMIA Annu Symp Proc"},{"issue":"(e2)","key":"2020110613001288500_R21","doi-asserted-by":"crossref","first-page":"e232","DOI":"10.1136\/amiajnl-2013-001932","article-title":"Defining a comprehensive verotype using electronic health records for personalized medicine","volume":"20","author":"Boland","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"(e2)","key":"2020110613001288500_R22","doi-asserted-by":"crossref","first-page":"e206","DOI":"10.1136\/amiajnl-2013-002428","article-title":"Electronic health records-driven phenotyping: challenges, recent advances, and perspectives","volume":"20","author":"Pathak","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R23","first-page":"14","article-title":"Network analysis of unstructured EHR data for clinical research","volume":"2013","author":"Bauer-Mehren","year":"2013","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"2020110613001288500_R24","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1186\/1546-0096-11-45","article-title":"Profiling risk factors for chronic uveitis in juvenile idiopathic arthritis: a new model for EHR-based research","volume":"11","author":"Cole","year":"2013","journal-title":"Pediatr Rheumatol Online J"},{"issue":"(Pt 2)","key":"2020110613001288500_R25","first-page":"758","article-title":"Extracting phenotypic information from the literature via natural language processing","volume":"107","author":"Chen","year":"2004","journal-title":"Stud Health Technol Inform"},{"key":"2020110613001288500_R26","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1136\/jamia.2009.001412","article-title":"Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC)","volume":"17","author":"D'Avolio","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R27","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R28","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","article-title":"Natural language processing: an introduction","volume":"18","author":"Nadkarni","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R29","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1186\/1471-2288-9-70","article-title":"Evaluating the informatics for integrating biology and the bedside system for clinical research","volume":"9","author":"Deshmukh","year":"2009","journal-title":"BMC Med Res Methodol"},{"key":"2020110613001288500_R30","first-page":"128","article-title":"Extracting information from textual documents in the electronic health record: a review of recent research","author":"Meystre","year":"2008","journal-title":"Yearb Med Inform"},{"issue":"(Web Server issue)","key":"2020110613001288500_R31","doi-asserted-by":"crossref","first-page":"W170","DOI":"10.1093\/nar\/gkp440","article-title":"BioPortal: ontologies and integrated data resources at the click of a mouse","volume":"37","author":"Noy","year":"2009","journal-title":"Nucleic Acids Res"},{"issue":"(Suppl 1)","key":"2020110613001288500_R32","doi-asserted-by":"crossref","first-page":"S5","DOI":"10.1186\/2041-1480-3-S1-S5","article-title":"Annotation analysis for testing drug safety signals using unstructured clinical notes","volume":"3","author":"Lependu","year":"2012","journal-title":"J Biomed Semantics"},{"key":"2020110613001288500_R33","first-page":"56","article-title":"The Open Biomedical Annotator","volume":"2009","author":"Jonquet","year":"2009","journal-title":"Summit on Translat Bioinforma"},{"key":"2020110613001288500_R34","first-page":"391","article-title":"STRIDE\u2013An integrated standards-based translational research informatics platform","volume":"2009","author":"Lowe","year":"2009","journal-title":"AMIA Annu Symp Proc"},{"issue":"(Database issue)","key":"2020110613001288500_R35","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2020110613001288500_R36","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1006\/jbin.2001.1029","article-title":"A simple algorithm for identifying negated findings and diseases in discharge summaries","volume":"34","author":"Chapman","year":"2001","journal-title":"J Biomed Inform"},{"issue":"(Suppl 9)","key":"2020110613001288500_R37","doi-asserted-by":"crossref","first-page":"S14","DOI":"10.1186\/1471-2105-10-S9-S14","article-title":"Comparison of concept recognizers for building the Open Biomedical Annotator","volume":"10","author":"Shah","year":"2009","journal-title":"BMC Bioinformatics"},{"key":"2020110613001288500_R38"},{"key":"2020110613001288500_R39","first-page":"270","article-title":"A broad-coverage natural language processing system","author":"Friedman","year":"2000","journal-title":"Proc AMIA Symp"},{"key":"2020110613001288500_R40","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1197\/jamia.M3115","article-title":"Recognizing obesity and comorbidities in sparse data","volume":"16","author":"Uzuner","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R41","doi-asserted-by":"crossref","first-page":"e63499","DOI":"10.1371\/journal.pone.0063499","article-title":"Practice-based evidence: profiling the safety of cilostazol by text-mining of clinical notes","volume":"8","author":"Leeper","year":"2013","journal-title":"PloS ONE"},{"key":"2020110613001288500_R42","first-page":"907","article-title":"A comprehensive analysis of five million UMLS metathesaurus terms using eighteen million MEDLINE citations","volume":"2010","author":"Xu","year":"2010","journal-title":"AMIA Annu Symp Proc"},{"issue":"(e1)","key":"2020110613001288500_R43","doi-asserted-by":"crossref","first-page":"e149","DOI":"10.1136\/amiajnl-2011-000744","article-title":"UMLS term occurrences in clinical notes: a large scale corpus analysis","volume":"19","author":"Wu","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R44","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1016\/j.jbi.2003.11.002","article-title":"Exploring semantic groups through visual approaches","volume":"36","author":"Bodenreider","year":"2003","journal-title":"J Biomed Inform"},{"key":"2020110613001288500_R45","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1136\/amiajnl-2011-000116","article-title":"Normalized names for clinical drugs: RxNorm at 6 years","volume":"18","author":"Nelson","year":"2011","journal-title":"J Am Med Inform Assoc Journal of the American"},{"key":"2020110613001288500_R46","author":"Health Fidelity"},{"key":"2020110613001288500_R47","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1038\/clpt.2013.47","article-title":"Pharmacovigilance using clinical notes","volume":"93","author":"Lependu","year":"2013","journal-title":"Clin Pharmacol Ther"},{"issue":"(Database issue)","key":"2020110613001288500_R48","doi-asserted-by":"crossref","first-page":"D1035","DOI":"10.1093\/nar\/gkq1126","article-title":"DrugBank 3.0: a comprehensive resource for \u2018omics\u2019 research on drugs","volume":"39","author":"Knox","year":"2011","journal-title":"Nucleic Acids Res"},{"key":"2020110613001288500_R49","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"2020110613001288500_R50","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R51","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1136\/amiajnl-2011-000093","article-title":"The Yale cTAKES extensions for document classification: architecture and application","volume":"18","author":"Garla","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613001288500_R52","doi-asserted-by":"crossref","first-page":"873","DOI":"10.1016\/j.jbi.2008.11.001","article-title":"Specializing for predicting obesity and its co-morbidities","volume":"42","author":"Goldstein","year":"2009","journal-title":"J Biomed Inform"},{"key":"2020110613001288500_R53","doi-asserted-by":"crossref","first-page":"1102","DOI":"10.1038\/nbt.2749","article-title":"Systematic comparison of phenome-wide association study of electronic medical record data and genome-wide association study data","volume":"31","author":"Denny","year":"2013","journal-title":"Nat Biotechnol"},{"key":"2020110613001288500_R54","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1038\/nrg3208","article-title":"Mining electronic health records: towards better research applications and clinical care","volume":"13","author":"Jensen","year":"2012","journal-title":"Nat Rev Genet"},{"key":"2020110613001288500_R55","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2013.393","article-title":"The inevitable application of big data to health care","volume":"309","author":"Murdoch","year":"2013","journal-title":"JAMA"},{"key":"2020110613001288500_R56","first-page":"848","article-title":"Automated identification of postoperative complications within an electronic medical record using natural language processing","volume":"306","author":"Murff","year":"2011","journal-title":"JAMA"},{"key":"2020110613001288500_R57","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.1038\/nbt.2757","article-title":"Mining the ultimate phenome repository","volume":"31","author":"Shah","year":"2013","journal-title":"Nat Biotechnol"},{"key":"2020110613001288500_R58","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MIS.2009.36","article-title":"The Unreasonable Effectiveness of Data","volume":"24","author":"Halevy","year":"2009","journal-title":"IEEE Intell Syst"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/1\/121\/34145305\/amiajnl-2014-002902.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/22\/1\/121\/34145305\/amiajnl-2014-002902.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,20]],"date-time":"2022-04-20T10:49:09Z","timestamp":1650451749000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/22\/1\/121\/834418"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,10,21]]},"references-count":58,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2014,10,21]]},"published-print":{"date-parts":[[2015,1,1]]}},"URL":"https:\/\/doi.org\/10.1136\/amiajnl-2014-002902","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,1]]},"published":{"date-parts":[[2014,10,21]]}}}