{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T21:35:30Z","timestamp":1790717730589,"version":"4.1.0"},"reference-count":70,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T00:00:00Z","timestamp":1712102400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/100007184","name":"Yale School of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007184","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Academic Emergency Medicine"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large\u2010scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large\u2010scale research, public health surveillance, and EHR\u2010based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>The study population consisted of patients aged \u2265\u200918 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task\u2010specific LLMs to perform the task of named entity recognition: a convolutional neural network\u2010based model (SpaCy) and a transformer\u2010based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note\u2010level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232\/250 correctly classified) for the SpaCy model and 0.98 (240\/250 correctly classified) for the Longformer model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The study demonstrated the utility of LLMs and transformer\u2010based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/acem.14883","type":"journal-article","created":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T05:59:31Z","timestamp":1712123971000},"page":"599-610","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models"],"prefix":"10.1111","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4446-2488","authenticated-orcid":false,"given":"Mark","family":"Iscoe","sequence":"first","affiliation":[{"name":"Department of Emergency Medicine Yale School of Medicine  New Haven Connecticut USA"},{"name":"Section for Biomedical Informatics and Data Science Yale University School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vimig","family":"Socrates","sequence":"additional","affiliation":[{"name":"Section for Biomedical Informatics and Data Science Yale University School of Medicine  New Haven Connecticut USA"},{"name":"Program of Computational Biology and Bioinformatics Yale University  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aidan","family":"Gilson","sequence":"additional","affiliation":[{"name":"Yale School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ling","family":"Chi","sequence":"additional","affiliation":[{"name":"Department of Biostatistics Yale School of Public Health  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huan","family":"Li","sequence":"additional","affiliation":[{"name":"Program of Computational Biology and Bioinformatics Yale University  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Huang","sequence":"additional","affiliation":[{"name":"Yale School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Kearns","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine Yale School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rachelle","family":"Perkins","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine Yale School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Khandjian","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine Yale School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9082-6644","authenticated-orcid":false,"given":"R. Andrew","family":"Taylor","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine Yale School of Medicine  New Haven Connecticut USA"},{"name":"Section for Biomedical Informatics and Data Science Yale University School of Medicine  New Haven Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2024,4,3]]},"reference":[{"key":"e_1_2_9_2_1","unstructured":"Office of the National Coordinator for Health Information Technology.National Trends in Hospital and Physician Adoption of Electronic Health Records. Health IT Quick\u2010Stat #61. HealthIT.gov2021."},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2022.100511"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-021-01347-1"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.2196\/12239"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.07.012"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2018.5097"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-022-13428-4"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2017.2767063"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocz200"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"e_1_2_9_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-023-00370-1"},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"e_1_2_9_14_1","author":"Gu Y","year":"2020","journal-title":"arXiv [csCL]"},{"key":"e_1_2_9_15_1","article-title":"Text chunking using transformation\u2010based learning [Internet]","author":"Ramshaw LA","year":"1995","journal-title":"arXiv [Cmp\u2010Lg]"},{"key":"e_1_2_9_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-81-322-3972-7_19"},{"key":"e_1_2_9_17_1","article-title":"Attention is all you need [Internet]","author":"Vaswani A","year":"2017","journal-title":"arXiv [csCL]"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/11941439_114"},{"key":"e_1_2_9_19_1","unstructured":"Antibiotic Resistance Threats in the United States.Centers for Disease Control and Prevention.2019. Accessed April 3 2023.https:\/\/stacks.cdc.gov\/view\/cdc\/82532"},{"issue":"169","key":"e_1_2_9_20_1","first-page":"1","article-title":"Ambulatory medical care utilization estimates for 2007","author":"Schappert SM","year":"2011","journal-title":"Vital Health Stat 13"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.emc.2018.06.012"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.emc.2018.06.003"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.annemergmed.2020.08.022"},{"key":"e_1_2_9_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.pop.2018.05.005"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/jgs.14912"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1128\/JCM.00670-15"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12873-020-00333-y"},{"issue":"8","key":"e_1_2_9_28_1","article-title":"Urinary tract infection as the diagnosis for admission through the emergency department: its prevalence, seasonality, diagnostic methods, and diagnostic decisions","volume":"14","author":"Alrashid S","year":"2022","journal-title":"Cureus"},{"key":"e_1_2_9_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2022.e11049"},{"issue":"12","key":"e_1_2_9_30_1","article-title":"Assessment of testing and treatment of asymptomatic bacteriuria initiated in the emergency department. Open forum","volume":"7","author":"Petty LA","year":"2020","journal-title":"Infect Dis"},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2016.0067"},{"key":"e_1_2_9_32_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2019.2871"},{"key":"e_1_2_9_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/jhm.2508"},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0194085"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamda.2020.12.010"},{"key":"e_1_2_9_36_1","doi-asserted-by":"publisher","DOI":"10.3389\/fcell.2020.00673"},{"key":"e_1_2_9_37_1","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M2408"},{"key":"e_1_2_9_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2007.11.008"},{"key":"e_1_2_9_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.idc.2013.09.003"},{"key":"e_1_2_9_40_1","doi-asserted-by":"publisher","DOI":"10.7326\/AITC201710030"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.1093\/cid\/ciq257"},{"key":"e_1_2_9_42_1","doi-asserted-by":"publisher","DOI":"10.1093\/cid\/ciz021"},{"key":"e_1_2_9_43_1","doi-asserted-by":"publisher","DOI":"10.1086\/501875"},{"key":"e_1_2_9_44_1","doi-asserted-by":"publisher","DOI":"10.3928\/0098-9134-20040601-04"},{"key":"e_1_2_9_45_1","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbz130"},{"key":"e_1_2_9_46_1","unstructured":"MontaniI HonnibalM.rodigy: A New Tool for Radically Efficient Machine Teaching. Explosion.2017. Accessed January 8 2024.https:\/\/explosion.ai\/blog\/prodigy\u2010annotation\u2010tool\u2010active\u2010learning"},{"key":"e_1_2_9_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3605943"},{"key":"e_1_2_9_48_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocac225"},{"key":"e_1_2_9_49_1","article-title":"Clinical\u2010Longformer and clinical\u2010BigBird: transformers for long clinical sequences [Internet]","author":"Li Y","year":"2022","journal-title":"arXiv [csCL]"},{"key":"e_1_2_9_50_1","article-title":"Longformer: the long\u2010document transformer [Internet]","author":"Beltagy I","year":"2020","journal-title":"arXiv [csCL]"},{"key":"e_1_2_9_51_1","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35"},{"key":"e_1_2_9_52_1","article-title":"A scikit\u2010based python environment for performing multi\u2010label classification [Internet]","author":"Szyma\u0144ski P","year":"2017","journal-title":"arXiv [csLG]"},{"key":"e_1_2_9_53_1","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M1733"},{"key":"e_1_2_9_54_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1009213286"},{"key":"e_1_2_9_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-021-02635-5"},{"key":"e_1_2_9_56_1","unstructured":"Jacob Gildenblat. 2023 Deriving confidence intervals for variants of recall precision and F1 [Internet]"},{"key":"e_1_2_9_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103354"},{"key":"e_1_2_9_58_1","first-page":"356","article-title":"General symptom extraction from VA electronic medical notes","volume":"245","author":"Divita G","year":"2017","journal-title":"Stud Health Technol Inform"},{"key":"e_1_2_9_59_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12631"},{"key":"e_1_2_9_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-33966-1_3"},{"key":"e_1_2_9_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2022.104279"},{"key":"e_1_2_9_62_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamaophthalmol.2016.5551"},{"key":"e_1_2_9_63_1","doi-asserted-by":"publisher","DOI":"10.4065\/80.2.203"},{"key":"e_1_2_9_64_1","doi-asserted-by":"publisher","DOI":"10.1093\/sleep\/zsad029"},{"key":"e_1_2_9_65_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa027"},{"key":"e_1_2_9_66_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.11390"},{"key":"e_1_2_9_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajic.2015.04.208"},{"key":"e_1_2_9_68_1","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6947-11-50"},{"key":"e_1_2_9_69_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0279953"},{"key":"e_1_2_9_70_1","doi-asserted-by":"publisher","DOI":"10.1111\/lnc3.12432"},{"key":"e_1_2_9_71_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2669-y"}],"container-title":["Academic Emergency Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/acem.14883","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,12]],"date-time":"2024-06-12T02:20:46Z","timestamp":1718158846000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/acem.14883"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,3]]},"references-count":70,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1111\/acem.14883"],"URL":"https:\/\/doi.org\/10.1111\/acem.14883","archive":["Portico"],"relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.10.20.23297156","asserted-by":"object"}]},"ISSN":["1069-6563","1553-2712"],"issn-type":[{"value":"1069-6563","type":"print"},{"value":"1553-2712","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,3]]},"assertion":[{"value":"2023-10-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-24","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}