{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T20:40:27Z","timestamp":1782765627246,"version":"3.54.5"},"reference-count":18,"publisher":"Springer Science and Business Media LLC","issue":"S11","license":[{"start":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2020,12,30]],"date-time":"2020-12-30T00:00:00Z","timestamp":1609286400000},"content-version":"vor","delay-in-days":29,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Eagles Charitable Foundation"},{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["LM012895"],"award-info":[{"award-number":["LM012895"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Foerderer Award"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Natural language processing (NLP) tools can facilitate the extraction of biomedical concepts from unstructured free texts, such as research articles or clinical notes. The NLP software tools CLAMP, cTAKES, and MetaMap are among the most widely used tools to extract biomedical concept entities. However, their performance in extracting disease-specific terminology from literature has not been compared extensively, especially for complex neuropsychiatric disorders with a diverse set of phenotypic and clinical manifestations.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We comparatively evaluated these NLP tools using autism spectrum disorder (ASD) as a case study. We collected 827 ASD-related terms based on previous literature as the benchmark list for performance evaluation. Then, we applied\u00a0CLAMP, cTAKES, and MetaMap on 544 full-text articles and 20,408 abstracts from PubMed to extract\u00a0ASD-related terms. We evaluated the predictive performance using precision, recall, and F1 score.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>We found that CLAMP has the best performance in terms of F1 score followed by cTAKES and then MetaMap. Our results show that CLAMP has much higher precision than cTAKES and MetaMap, while cTAKES and MetaMap have higher recall than CLAMP.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>The analysis protocols used in this study can be applied to other neuropsychiatric or neurodevelopmental disorders that lack well-defined terminology sets to describe their phenotypic presentations.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-020-01352-2","type":"journal-article","created":{"date-parts":[[2020,12,30]],"date-time":"2020-12-30T07:02:24Z","timestamp":1609311744000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Natural language processing (NLP) tools in extracting biomedical concepts from research articles: a case study on autism spectrum disorder"],"prefix":"10.1186","volume":"20","author":[{"given":"Jacqueline","family":"Peng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mengge","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Havrilla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cong","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Whitney","family":"Guthrie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Schultz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kai","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1689-8575","authenticated-orcid":false,"given":"Yunyun","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,12,30]]},"reference":[{"issue":"3","key":"1352_CR1","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1136\/jamia.2009.002733","volume":"17","author":"AR Aronson","year":"2010","unstructured":"Aronson AR, Lang F-M. An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc. 2010;17(3):229\u201336.","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"1352_CR2","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc. 2010;17(5):507\u201313.","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"1352_CR3","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1093\/jamia\/ocx132","volume":"25","author":"E Soysal","year":"2018","unstructured":"Soysal E, Wang J, Jiang M, Wu Y, Pakhomov S, Liu H, Xu H. CLAMP\u2013a toolkit for efficiently building customized clinical natural language processing pipelines. J Am Med Inform Assoc. 2018;25(3):331\u20136.","journal-title":"J Am Med Inform Assoc"},{"issue":"e1","key":"1352_CR4","doi-asserted-by":"publisher","first-page":"e149","DOI":"10.1136\/amiajnl-2011-000744","volume":"19","author":"ST Wu","year":"2012","unstructured":"Wu ST, Liu H, Li D, Tao C, Musen MA, Chute CG, Shah NH. Unified Medical Language System term occurrences in clinical notes: a large-scale corpus analysis. J Am Med Inform Assoc. 2012;19(e1):e149\u201356.","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"1352_CR5","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1186\/s12911-018-0654-2","volume":"18","author":"R Re\u00e1tegui","year":"2018","unstructured":"Re\u00e1tegui R, Ratt\u00e9 S. Comparison of MetaMap and cTAKES for entity extraction in clinical notes. BMC Med Inform Decis Mak. 2018;18(3):74.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"6","key":"1352_CR6","doi-asserted-by":"publisher","first-page":"821","DOI":"10.1089\/106652703322756104","volume":"10","author":"H Shatkay","year":"2003","unstructured":"Shatkay H, Feldman R. Mining the biomedical literature in the genomic era: an overview. J Comput Biol. 2003;10(6):821\u201355.","journal-title":"J Comput Biol"},{"issue":"1","key":"1352_CR7","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1001\/jama.2017.17812","volume":"319","author":"G Xu","year":"2018","unstructured":"Xu G, Strathearn L, Liu B, Bao W. Prevalence of autism spectrum disorder among US children and adolescents, 2014\u20132016. JAMA. 2018;319(1):81\u20132.","journal-title":"JAMA"},{"issue":"19","key":"1352_CR8","doi-asserted-by":"publisher","first-page":"R786","DOI":"10.1016\/j.cub.2005.09.033","volume":"15","author":"U Frith","year":"2005","unstructured":"Frith U, Happ\u00e9 F. Autism spectrum disorder. Curr Biol. 2005;15(19):R786\u201390.","journal-title":"Curr Biol"},{"issue":"e2","key":"1352_CR9","doi-asserted-by":"publisher","first-page":"e297","DOI":"10.1136\/amiajnl-2013-001933","volume":"20","author":"S Lyalina","year":"2013","unstructured":"Lyalina S, Percha B, LePendu P, Iyer SV, Altman RB, Shah NH. Identifying phenotypic signatures of neuropsychiatric disorders from electronic medical records. J Am Med Inform Assoc. 2013;20(e2):e297\u2013305.","journal-title":"J Am Med Inform Assoc"},{"issue":"7","key":"1352_CR10","doi-asserted-by":"publisher","first-page":"e0159621","DOI":"10.1371\/journal.pone.0159621","volume":"11","author":"T Lingren","year":"2016","unstructured":"Lingren T, Chen P, Bochenek J, Doshi-Velez F, Manning-Courtney P, Bickel J, Welchons LW, Reinhold J, Bing N, Ni Y. Electronic health record based algorithm to identify patients with autism spectrum disorder. PLoS ONE. 2016;11(7):e0159621.","journal-title":"PLoS ONE"},{"issue":"1","key":"1352_CR11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1001\/archpedi.159.1.37","volume":"159","author":"WJ Barbaresi","year":"2005","unstructured":"Barbaresi WJ, Katusic SK, Colligan RC, Weaver AL, Jacobsen SJ. The Incidence of Autism in Olmsted County, Minnesota, 1976\u20131997: results from a population-based study. Arch Pediatr Adolesc Med. 2005;159(1):37\u201344.","journal-title":"Arch Pediatr Adolesc Med."},{"issue":"7","key":"1352_CR12","doi-asserted-by":"publisher","first-page":"849","DOI":"10.1007\/s10803-006-0123-0","volume":"36","author":"OT Leyfer","year":"2006","unstructured":"Leyfer OT, Folstein SE, Bacalman S, Davis NO, Dinh E, Morgan J, Tager-Flusberg H, Lainhart JE. Comorbid psychiatric disorders in children with autism: interview development and rates of disorders. J Autism Dev Disord. 2006;36(7):849\u201361.","journal-title":"J Autism Dev Disord"},{"issue":"1","key":"1352_CR13","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1177\/0883073807307102","volume":"23","author":"X Ming","year":"2008","unstructured":"Ming X, Brimacombe M, Chaaban J, Zimmerman-Bier B, Wagner GC. Autism spectrum disorders: concurrent clinical disorders. J Child Neurol. 2008;23(1):6\u201313.","journal-title":"J Child Neurol"},{"issue":"1","key":"1352_CR14","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1186\/2040-2392-4-12","volume":"4","author":"R Grzadzinski","year":"2013","unstructured":"Grzadzinski R, Huerta M, Lord C. DSM-5 and autism spectrum disorders (ASDs): an opportunity for identifying ASD subtypes. Molecular autism. 2013;4(1):12.","journal-title":"Molecular autism"},{"issue":"e2","key":"1352_CR15","doi-asserted-by":"publisher","first-page":"e355","DOI":"10.1136\/amiajnl-2013-001946","volume":"20","author":"AV Gundlapalli","year":"2013","unstructured":"Gundlapalli AV, Redd A, Carter M, Divita G, Shen S, Palmer M, Samore MH. Validating a strategy for psychosocial phenotyping using a large corpus of clinical text. J Am Med Inform Assoc. 2013;20(e2):e355\u201364.","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"1352_CR16","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1007\/s40474-013-0003-1","volume":"1","author":"O Ousley","year":"2014","unstructured":"Ousley O, Cermak T. Autism spectrum disorder: defining dimensions and subgroups. Curr Dev Disord Rep. 2014;1(1):20\u20138.","journal-title":"Curr Dev Disord Rep"},{"issue":"2","key":"1352_CR17","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1111\/j.1469-7610.1997.tb01855.x","volume":"38","author":"S Ehlers","year":"1997","unstructured":"Ehlers S, Nyd\u00e9n A, Gillberg C, Sandberg AD, Dahlgren SO, Hjelmquist E, Od\u00e9n A. Asperger syndrome, autism and attention disorders: A comparative study of the cognitive profiles of 120 children. J Child Psychol Psychiatry. 1997;38(2):207\u201317.","journal-title":"J Child Psychol Psychiatry"},{"issue":"e1","key":"1352_CR18","doi-asserted-by":"publisher","first-page":"e79","DOI":"10.1093\/jamia\/ocw109","volume":"24","author":"Y Wu","year":"2017","unstructured":"Wu Y, Denny JC, Trent Rosenbloom S, Miller RA, Giuse DA, Wang L, Blanquicett C, Soysal E, Xu J, Xu H. A long journey to short abbreviations: developing an open-source framework for clinical abbreviation recognition and disambiguation (CARD). J Am Med Inform Assoc. 2017;24(e1):e79\u201386.","journal-title":"J Am Med Inform Assoc"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01352-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12911-020-01352-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01352-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,30]],"date-time":"2020-12-30T07:02:47Z","timestamp":1609311767000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-01352-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":18,"journal-issue":{"issue":"S11","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1352"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-01352-2","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]},"assertion":[{"value":"23 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 November 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 December 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare there is no conflict of interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"322"}}