{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T06:44:45Z","timestamp":1792478685188,"version":"4.5.0"},"reference-count":81,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T00:00:00Z","timestamp":1670457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:p>Although deep learning has been applied to the recognition of diseases and drugs in electronic health records and the biomedical literature, relatively little study has been devoted to the utility of deep learning for the recognition of signs and symptoms. The recognition of signs and symptoms is critical to the success of deep phenotyping and precision medicine. We have developed a named entity recognition model that uses deep learning to identify text spans containing neurological signs and symptoms and then maps these text spans to the clinical concepts of a neuro-ontology. We compared a model based on convolutional neural networks to one based on bidirectional encoder representation from transformers. Models were evaluated for accuracy of text span identification on three text corpora: physician notes from an electronic health record, case histories from neurologic textbooks, and clinical synopses from an online database of genetic diseases. Both models performed best on the professionally-written clinical synopses and worst on the physician-written clinical notes. Both models performed better when signs and symptoms were represented as shorter text spans. Consistent with prior studies that examined the recognition of diseases and drugs, the model based on bidirectional encoder representations from transformers outperformed the model based on convolutional neural networks for recognizing signs and symptoms. Recall for signs and symptoms ranged from 59.5% to 82.0% and precision ranged from 61.7% to 80.4%. With further advances in NLP, fully automated recognition of signs and symptoms in electronic health records and the medical literature should be feasible.<\/jats:p>","DOI":"10.3389\/fdgth.2022.1065581","type":"journal-article","created":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T05:00:50Z","timestamp":1670475650000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Enhanced neurologic concept recognition using a named entity recognition model based on transformers"],"prefix":"10.3389","volume":"4","author":[{"given":"Sima","family":"Azizi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel B.","family":"Hier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donald C.","family":"Wunsch II","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,12,8]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1016\/j.jbi.2004.08.004","article-title":"Term identification in the biomedical literature","volume":"37","author":"Krauthammer","year":"2004","journal-title":"J Biomed Inform"},{"key":"B2","year":""},{"key":"B3","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1146\/annurev-biodatasci-080917-013315","article-title":"Advances in electronic phenotyping: from rule-based definitions to machine learning models","volume":"1","author":"Banda","year":"2018","journal-title":"Annu Rev Biomed Data Sci"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1045\/may2003-king","article-title":"Patterns of journal use by scientists through three evolutionary phases","author":"Tenopir","year":"2003","journal-title":"D-Lib"},{"key":"B5","doi-asserted-by":"publisher","first-page":"793","DOI":"10.1056\/NEJMp1500523","article-title":"A new initiative on precision medicine","volume":"372","author":"Collins","year":"2015","journal-title":"N Engl J Med"},{"key":"B6","doi-asserted-by":"publisher","first-page":"777","DOI":"10.1002\/humu.22080","article-title":"Deep phenotyping for precision medicine","volume":"33","author":"Robinson","year":"2012","journal-title":"Hum Mutat"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103526","article-title":"Clinical concept extraction: a methodology review","author":"Fu","year":"","journal-title":"J Biomed Inform"},{"key":"B8","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","article-title":"A guide to deep learning in healthcare","volume":"25","author":"Esteva","year":"2019","journal-title":"Nat Med"},{"key":"B9","first-page":"603","article-title":"Natural language processing","author":"Chowdhary","year":"2020","journal-title":"Fundam Artif Intell"},{"key":"B10","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1126\/science.aaa8685","article-title":"Advances in natural language processing","volume":"349","author":"Hirschberg","year":"2015","journal-title":"Science"},{"key":"B11","doi-asserted-by":"publisher","first-page":"12141","DOI":"10.1007\/s00521-021-05895-x","article-title":"Towards achieving a delicate blending between rule-based translator, neural machine translator","volume":"33","year":"2021","journal-title":"Neural Comput Appl"},{"key":"B12","author":"Islam","year":""},{"key":"B13","author":"Mohammad","year":""},{"key":"B14","author":"Hasan","year":""},{"key":"B15","doi-asserted-by":"publisher","first-page":"214","DOI":"10.15265\/IY-2017-029","article-title":"Capturing the patient\u2019s perspective: a review of advances in natural language processing of health-related text","volume":"26","author":"Gonzalez-Hernandez","year":"2017","journal-title":"Yearb Med Inform"},{"key":"B16","article-title":"Natural language processing with python","author":"Bird","year":"2009"},{"key":"B17","year":""},{"key":"B18","year":""},{"key":"B19","year":""},{"key":"B20","year":""},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1758-2946-6-17","article-title":"Chemical named entities recognition: a review on approaches, applications","volume":"6","author":"Eltyeb","year":"2014","journal-title":"J Cheminform"},{"key":"B22","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/j.procs.2016.09.123","article-title":"Named entity recognition over electronic health records through a combined dictionary-based approach","volume":"100","author":"Quimbaya","year":"2016","journal-title":"Procedia Comput Sci"},{"key":"B23","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1016\/S1532-0464(03)00014-5","article-title":"Rutabaga by any other name: extracting biological names","volume":"35","author":"Hirschman","year":"2002","journal-title":"J Biomed Inform"},{"key":"B24","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions,, relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"B25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-15-59","article-title":"Large-scale biomedical concept recognition: an evaluation of current automatic annotators and their parameters","volume":"15","author":"Funk","year":"2014","journal-title":"BMC Bioinf"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2105-10-S2-S1","article-title":"Comparison of concept recognizers for building the open biomedical annotator","volume":"10","author":"Shah","year":"2009","journal-title":"BMC Bioinf"},{"key":"B27","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"B28","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1055\/s-0038-1637976","article-title":"The unified medical language system","volume":"2","author":"Lindberg","year":"1993","journal-title":"Yearb Med Inform"},{"key":"B29","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"B30","doi-asserted-by":"publisher","first-page":"37","DOI":"10.19044\/esj.2022.v18n4p37","article-title":"High throughput neurological phenotyping with MetaMap","volume":"18","author":"Hier","year":"2022","journal-title":"Eur Sci J"},{"key":"B31","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"B32","doi-asserted-by":"publisher","first-page":"240","DOI":"10.1007\/s10278-011-9411-0","article-title":"Automatically correlating clinical findings and body locations in radiology reports using MedLEE","volume":"25","author":"Sevenster","year":"2012","journal-title":"J Digit Imaging"},{"key":"B33","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"B34","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1197\/jamia.M1552","article-title":"Automated encoding of clinical documents based on natural language processing","volume":"11","author":"Friedman","year":"2004","journal-title":"J Am Med Inform Assoc"},{"key":"B35","author":"Friedman","year":""},{"key":"B36","author":"Friedman","year":""},{"key":"B37","author":"Huang","year":""},{"key":"B38","author":"Lample","year":""},{"key":"B39","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1162\/tacl-a-00104","article-title":"Named entity recognition with bidirectional LSTM-CNNs","volume":"4","author":"Chiu","year":"2016","journal-title":"Trans Assoc Comput Linguist"},{"key":"B40","author":"Peters","year":""},{"key":"B41","author":"Vani","year":""},{"key":"B42","doi-asserted-by":"publisher","first-page":"S34","DOI":"10.1016\/j.jbi.2017.05.023","article-title":"De-identification of clinical notes via recurrent neural network and conditional random field","volume":"75","author":"Liu","year":"2017","journal-title":"J Biomed Inform"},{"key":"B43","doi-asserted-by":"publisher","first-page":"i37","DOI":"10.1093\/bioinformatics\/btx228","article-title":"Deep learning with word embeddings improves biomedical named entity recognition","volume":"33","author":"Habibi","year":"2017","journal-title":"Bioinformatics"},{"key":"B44","doi-asserted-by":"publisher","first-page":"S53","DOI":"10.1016\/j.jbi.2015.06.029","article-title":"Combining knowledge-and data-driven methods for de-identification of clinical narratives","volume":"58","author":"Dehghan","year":"2015","journal-title":"J Biomed Inform"},{"key":"B45","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"B46","author":"Lafferty","year":""},{"key":"B47","doi-asserted-by":"publisher","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","article-title":"Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives","volume":"13","author":"Gehrmann","year":"2018","journal-title":"PLoS ONE"},{"key":"B48","doi-asserted-by":"publisher","first-page":"e12596","DOI":"10.2196\/12596","article-title":"Identifying clinical terms in medical text using ontology-guided machine learning","volume":"7","author":"Arbabi","year":"2019","journal-title":"JMIR Med Inform"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1093\/database\/bav005","article-title":"Automatic concept recognition using the human phenotype ontology reference and test suite corpora","volume":"2015","author":"Groza","year":"2015","journal-title":"Database"},{"key":"B50","article-title":"Attention is all you need","author":"Vaswani","year":"","journal-title":"Adv Neural Inf Process Syst"},{"key":"B51","author":"Devlin","year":""},{"key":"B52","author":"Zhu","year":""},{"key":"B53","author":"Yu","year":""},{"key":"B54","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"B55","first-page":"269","article-title":"Bert-based ranking for biomedical entity normalization","volume":"2020","author":"Ji","year":"2020","journal-title":"AMIA Summits Transl Sci Proc"},{"key":"B56","doi-asserted-by":"publisher","first-page":"103433","DOI":"10.1016\/j.jbi.2020.103433","article-title":"Deep phenotyping: embracing complexity and temporality-towards scalability, portability, and interoperability","volume":"105","author":"Weng","year":"2020","journal-title":"J Biomed Inform"},{"key":"B57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12911-020-1066-7","article-title":"A neuro-ontology for the neurological examination","volume":"20","author":"Hier","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"B58","volume-title":"Neurology study guide: oral board examination review","author":"Gondolo","year":"2005"},{"key":"B59","volume-title":"Neurology oral boards review","author":"Ubogu","year":"2005"},{"key":"B60","volume-title":"The neurologic diagnosis: a practical bedside approach","author":"Alpert","year":"2018"},{"key":"B61","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-11132-8","volume-title":"Absolute case-based neurology review","author":"Kung","year":"2019"},{"key":"B62","volume-title":"Neurology clinical cases uncovered","author":"Macleod","year":"2011"},{"key":"B63","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1093\/bib\/bbz130","article-title":"An extensive review of tools for manual annotation of documents","volume":"22","author":"Neves","year":"2021","journal-title":"Brief Bioinformatics"},{"key":"B64","article-title":"Prodigy: a new annotation tool for radically efficient machine teaching","author":"Montani","year":"2018","journal-title":"Artif Intell"},{"key":"B65","author":"Oommen","year":""},{"key":"B66","volume-title":"Natural language processing with Python and Spacy","author":"Vasiliev","year":"2020"},{"key":"B67","author":"Noy","year":""},{"key":"B68","doi-asserted-by":"publisher","first-page":"66","DOI":"10.3389\/fmed.2019.00066","article-title":"The revival of the notes field: leveraging the unstructured content in electronic health records","volume":"6","author":"Assale","year":"2019","journal-title":"Front Med"},{"key":"B69","doi-asserted-by":"publisher","first-page":"82","DOI":"10.5116\/ijme.534a.a3e2","article-title":"Writing style of young physicians in the computer and internet era","volume":"5","author":"Shilo","year":"2014","journal-title":"Int J Med Educ"},{"key":"B70","doi-asserted-by":"publisher","first-page":"331","DOI":"10.2190\/WY9T-634E-V2JT-JDVQ","article-title":"Writing medical records","volume":"16","author":"Pagano","year":"1986","journal-title":"J Tech Writ Commun"},{"key":"B71","first-page":"481","article-title":"Teaching medical students and physicians to write","volume":"39","author":"Zisowitz","year":"1964","journal-title":"Acad Med"},{"key":"B72","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1136\/postgradmedj-2017-135515","article-title":"Frequency, comprehension and attitudes of physicians towards abbreviations in the medical\u00a0record","volume":"94","author":"Hamiel","year":"2018","journal-title":"Postgrad Med J"},{"key":"B73","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1136\/jamia.2010.007237","article-title":"Data from clinical notes: a perspective on the tension between structure and flexible documentation","volume":"18","author":"Rosenbloom","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"B74","doi-asserted-by":"publisher","first-page":"985","DOI":"10.1093\/jamia\/ocaa301","article-title":"The burden of the digital environment: a systematic review on organization-directed workplace interventions to mitigate physician burnout","volume":"28","author":"Thomas Craig","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"B75","doi-asserted-by":"publisher","first-page":"18634","DOI":"10.3402\/meo.v18i0.18634","article-title":"Writing and reading in the electronic health record: an entirely new world","volume":"18","author":"Han","year":"2013","journal-title":"Med Educ Online"},{"key":"B76","author":"Shivade","year":""},{"key":"B77","doi-asserted-by":"publisher","first-page":"e112774","DOI":"10.1371\/journal.pone.0112774","article-title":"Negation\u2019s not solved: generalizability versus optimizability in clinical natural language processing","volume":"9","author":"Wu","year":"2014","journal-title":"PLoS ONE"},{"key":"B78","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6947-5-13","article-title":"A controlled trial of automated classification of negation from clinical notes","volume":"5","author":"Elkin","year":"2005","journal-title":"BMC Med Inform Decis Mak"},{"key":"B79","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1459352.1459355","article-title":"Word sense disambiguation: a survey","volume":"41","author":"Navigli","year":"2009","journal-title":"ACM Comput Surv"},{"key":"B80","first-page":"58","article-title":"A survey of automated text simplification","volume":"4","author":"Shardlow","year":"2014","journal-title":"Int J Adv Comput Sci Appl"},{"key":"B81","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3442695","article-title":"Automated text simplification: a survey","volume":"54","author":"Al-Thanyyan","year":"2021","journal-title":"ACM Comput Surv"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2022.1065581\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T13:03:04Z","timestamp":1678971784000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2022.1065581\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,8]]},"references-count":81,"alternative-id":["10.3389\/fdgth.2022.1065581"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2022.1065581","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,8]]},"article-number":"1065581"}}