{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:06:12Z","timestamp":1792785972760,"version":"4.5.0"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2026,10,11]],"date-time":"2026-10-11T00:00:00Z","timestamp":1791676800000},"content-version":"vor","delay-in-days":32,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"National Institutes of Health (NIH) National Institute of General Medical Sciences","award":["R35 GM145330"],"award-info":[{"award-number":["R35 GM145330"]}],"id":[{"id":"https:\/\/ror.org\/04q48ey07","id-type":"ROR","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["T32 AG 74937"],"award-info":[{"award-number":["T32 AG 74937"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,9,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Expert clinician chart review is the most accurate method to identify patients with infection but is laborious and costly. We aimed to develop and externally validate machine-learning models using structured and unstructured data to identify infection in hospitalized patients compared to chart review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Adult inpatient encounters at 2 health systems (2006-2016) were manually annotated for suspected infection on admission. Data from one site were used to develop gradient-boosted machine-learning models. Structured features included labs, vital signs, and nursing assessments. Unstructured clinical notes were converted to concept unique identifiers (CUIs) using the clinical Text Analysis and Knowledge Extraction System. Unimodal and multimodal models were tuned to maximize the area under the receiver operating curve (AUROC) in the derivation site through cross-validation. Model discrimination and calibration were assessed in the external validation site.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Of the 3014 included admissions, 677 (22.5%) were classified as having suspected infection on admission. A multimodal model combining n-gram TF-IDF and structured data demonstrated the best performance, significantly outperforming the structured data only model (AUROC 0.88 vs 0.82, P\u2009&amp;lt;\u2009.001, area under the precision-recall curve 0.71 vs 0.58, P\u2009&amp;lt;\u2009.001). The most important features came from unstructured data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion and Conclusion<\/jats:title>\n                    <jats:p>Our results highlight the value of unstructured text data for developing models that accurately replicate chart reviews in identifying patients with infection. By expediting the annotation of infection cases, our models can facilitate epidemiological research and the curation of datasets necessary for developing clinical decision support systems for early sepsis detection.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag163","type":"journal-article","created":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T12:07:35Z","timestamp":1785586055000},"source":"Crossref","is-referenced-by-count":0,"title":["Automating chart review for identification of infected patients using natural language processing"],"prefix":"10.1093","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3108-4385","authenticated-orcid":false,"given":"Afia Z","family":"Khan","sequence":"first","affiliation":[{"name":"Pritzker School of Medicine, University of Chicago , Chicago, IL 60637,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin G","family":"Buell","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Chicago , Chicago, IL 60637,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George","family":"Karway","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle","family":"Carey","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Chicago , Chicago, IL 60637,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicole","family":"Dussault","sequence":"additional","affiliation":[{"name":"Department of Medicine, Duke University , Durham, NC 27710,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jay","family":"Dumanian","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Washington , Seattle, WA 98195,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sivasubramanium","family":"Bhavani","sequence":"additional","affiliation":[{"name":"Department of Medicine, Emory University , Atlanta, GA 30322,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily R","family":"Gilbert","sequence":"additional","affiliation":[{"name":"Department of Medicine, Loyola University , Chicago, IL 60153,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Winslow","sequence":"additional","affiliation":[{"name":"NorthShore University HealthSystem Department of Medicine, , Evanston, IL 60201,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9107-7788","authenticated-orcid":false,"given":"Nirav","family":"Shah","sequence":"additional","affiliation":[{"name":"NorthShore University HealthSystem Department of Medicine, , Evanston, IL 60201,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Majid","family":"Afshar","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792,","place":["United States"]},{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dana P","family":"Edelson","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Chicago , Chicago, IL 60637,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew M","family":"Churpek","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792,","place":["United States"]},{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison , Madison, WI 53792,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,10,11]]},"reference":[{"key":"2026101108341593600_ooag163-B1","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1001\/jama.2016.0287","article-title":"The third international consensus definitions for sepsis and septic shock (sepsis-3)","volume":"315","author":"Singer","year":"2016","journal-title":"JAMA"},{"key":"2026101108341593600_ooag163-B2","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1001\/jama.2014.5804","article-title":"Hospital deaths in patients with sepsis from 2 independent cohorts","volume":"312","author":"Liu","year":"2014","journal-title":"JAMA"},{"key":"2026101108341593600_ooag163-B3","doi-asserted-by":"publisher","first-page":"856","DOI":"10.1164\/rccm.201609-1848OC","article-title":"The timing of early antibiotics and hospital mortality in sepsis","volume":"196","author":"Liu","year":"2017","journal-title":"Am J Respir Crit Care Med"},{"key":"2026101108341593600_ooag163-B4","doi-asserted-by":"publisher","first-page":"929","DOI":"10.1016\/j.chest.2020.03.087","article-title":"A systematic review of the effect of delayed appropriate antibiotic treatment on the outcomes of patients with severe bacterial infections","volume":"158","author":"Zasowski","year":"2020","journal-title":"Chest"},{"key":"2026101108341593600_ooag163-B5","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1016\/S0140-6736(15)00474-2","article-title":"Access to effective antimicrobials: a worldwide challenge","volume":"387","author":"Laxminarayan","year":"2016","journal-title":"Lancet"},{"key":"2026101108341593600_ooag163-B6","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1177\/2042098614554919","article-title":"Antimicrobial resistance: risk associated with antibiotic overuse and initiatives to reduce the problem","volume":"5","author":"Llor","year":"2014","journal-title":"Ther Adv Drug Saf"},{"key":"2026101108341593600_ooag163-B7","doi-asserted-by":"publisher","first-page":"256","DOI":"10.12788\/jhm.2721","article-title":"Detecting sepsis: are two opinions better than one?","volume":"12","author":"Bhattacharjee","year":"2017","journal-title":"J Hosp Med"},{"key":"2026101108341593600_ooag163-B8","doi-asserted-by":"publisher","first-page":"898","DOI":"10.1016\/j.chest.2016.06.020","article-title":"Identifying patients with sepsis on the hospital wards","volume":"151","author":"Bhattacharjee","year":"2017","journal-title":"Chest"},{"key":"2026101108341593600_ooag163-B9","doi-asserted-by":"publisher","first-page":"1368","DOI":"10.1007\/s00134-003-1874-0","article-title":"The systemic inflammatory response syndrome (SIRS) to identify infected patients in the emergency room","volume":"29","author":"Jaimes","year":"2003","journal-title":"Intensive Care Med"},{"key":"2026101108341593600_ooag163-B10","doi-asserted-by":"publisher","first-page":"8095","DOI":"10.1038\/s41598-020-64314-8","article-title":"The SIRS criteria have better performance for predicting infection than qSOFA scores in the emergency department","volume":"10","author":"Gando","year":"2020","journal-title":"Sci Rep"},{"key":"2026101108341593600_ooag163-B11","doi-asserted-by":"publisher","first-page":"e673","DOI":"10.1097\/CCM.0000000000004968","article-title":"Determining the electronic signature of infection in electronic health record data","volume":"49","author":"Churpek","year":"2021","journal-title":"Crit Care Med"},{"key":"2026101108341593600_ooag163-B12","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1093\/jamia\/ocab236","article-title":"Sepsis prediction, early detection, and identification using clinical text for machine learning: a systematic review","volume":"29","author":"Yan","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2026101108341593600_ooag163-B13","doi-asserted-by":"publisher","first-page":"6103","DOI":"10.1109\/EMBC.2019.8857819","article-title":"Natural language processing of clinical notes for improved early prediction of septic shock in the ICU","volume":"2019","author":"Liu","year":"2019","journal-title":"Annu Int Conf IEEE Eng Med Biol Soc"},{"key":"2026101108341593600_ooag163-B14","first-page":"197","article-title":"Contextual embeddings from clinical notes improves prediction of sepsis","volume":"2020","author":"Amrollahi","year":"2020","journal-title":"AMIA Annu Symp Proc"},{"key":"2026101108341593600_ooag163-B15","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1016\/j.annemergmed.2018.11.036","article-title":"Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis","volume":"73","author":"Delahanty","year":"2019","journal-title":"Ann Emerg Med"},{"key":"2026101108341593600_ooag163-B16","doi-asserted-by":"publisher","first-page":"711","DOI":"10.1038\/s41467-021-20910-4","article-title":"Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare","volume":"12","author":"Goh","year":"2021","journal-title":"Nat Commun"},{"key":"2026101108341593600_ooag163-B17","doi-asserted-by":"publisher","first-page":"e0174708","DOI":"10.1371\/journal.pone.0174708","article-title":"Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning","volume":"12","author":"Horng","year":"2017","journal-title":"PLoS One"},{"key":"2026101108341593600_ooag163-B18","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1016\/j.ajem.2024.03.006","article-title":"Early identification of suspected serious infection among patients afebrile at initial presentation using neural network models and natural language processing: a development and external validation study in the emergency department","volume":"80","author":"Choi","year":"2024","journal-title":"Am J Emerg Med"},{"key":"2026101108341593600_ooag163-B19","doi-asserted-by":"publisher","first-page":"257","DOI":"10.18653\/v1\/W17-2332","volume-title":"BioNLP 2017","author":"Apostolova","year":"2017"},{"key":"2026101108341593600_ooag163-B20","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1186\/s12874-022-01583-z","article-title":"Automated medical chart review for breast cancer outcomes research: a novel natural language processing extraction system","volume":"22","author":"Chen","year":"2022","journal-title":"BMC Med Res Methodol"},{"key":"2026101108341593600_ooag163-B21","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1016\/j.anai.2013.07.022","article-title":"Automated chart review for asthma cohort identification using natural language processing: an exploratory study","volume":"111","author":"Wu","year":"2013","journal-title":"Ann Allergy Asthma Immunol"},{"key":"2026101108341593600_ooag163-B22","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1093\/aje\/kwt441","article-title":"Using natural language processing to improve efficiency of manual chart abstraction in research: the case of breast cancer recurrence","volume":"179","author":"Carrell","year":"2014","journal-title":"Am J Epidemiol"},{"key":"2026101108341593600_ooag163-B23","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1186\/s40708-023-00203-w","article-title":"Cerebrovascular disease case identification in inpatient electronic medical record data using natural language processing","volume":"10","author":"Pan","year":"2023","journal-title":"Brain Inform"},{"key":"2026101108341593600_ooag163-B24","doi-asserted-by":"publisher","first-page":"1303","DOI":"10.1097\/00003246-200107000-00002","article-title":"Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care","volume":"29","author":"Angus","year":"2001","journal-title":"Crit Care Med"},{"key":"2026101108341593600_ooag163-B25","doi-asserted-by":"publisher","first-page":"e39","DOI":"10.1097\/MLR.0b013e318268ac86","article-title":"Identifying patients with severe sepsis using administrative claims: patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis","volume":"52","author":"Iwashyna","year":"2014","journal-title":"Med Care"},{"key":"2026101108341593600_ooag163-B26","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2026101108341593600_ooag163-B27","author":"UW ICU Data Science Lab","year":"2025"},{"key":"2026101108341593600_ooag163-B28","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res."},{"key":"2026101108341593600_ooag163-B29","doi-asserted-by":"publisher","first-page":"ooad109","DOI":"10.1093\/jamiaopen\/ooad109","article-title":"Development and external validation of multimodal postoperative acute kidney injury risk machine learning models","volume":"6","author":"Karway","year":"2023","journal-title":"JAMIA Open"},{"key":"2026101108341593600_ooag163-B30","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1145\/2939672.2939785","author":"Chen","year":"2016"},{"key":"2026101108341593600_ooag163-B31","doi-asserted-by":"publisher","author":"Snoek","year":"2012","DOI":"10.48550\/arXiv.1206.2944"},{"key":"2026101108341593600_ooag163-B32","doi-asserted-by":"publisher","first-page":"1644","DOI":"10.1378\/chest.101.6.1644","article-title":"Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis","volume":"101","author":"Bone","year":"1992","journal-title":"Chest"},{"key":"2026101108341593600_ooag163-B33","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143874"},{"key":"2026101108341593600_ooag163-B34","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"2026101108341593600_ooag163-B35","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1093\/jamia\/ocz228","article-title":"A tutorial on calibration measurements and calibration models for clinical prediction models","volume":"27","author":"Huang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2026101108341593600_ooag163-B36","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1001\/jama.2016.0288","article-title":"Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3)","volume":"315","author":"Seymour","year":"2016","journal-title":"JAMA"},{"key":"2026101108341593600_ooag163-B37","doi-asserted-by":"publisher","first-page":"1241","DOI":"10.1001\/jama.2017.13836","article-title":"Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014","volume":"318","author":"Rhee","year":"2017","journal-title":"JAMA"},{"key":"2026101108341593600_ooag163-B38","doi-asserted-by":"publisher","first-page":"1686","DOI":"10.21105\/joss.01686","article-title":"Welcome to the tidyverse","volume":"4","author":"Wickham","year":"2019","journal-title":"J Open Source Softw"},{"key":"2026101108341593600_ooag163-B39","doi-asserted-by":"publisher","first-page":"37","DOI":"10.21105\/joss.00037","article-title":"Tidytext: Text mining and analysis using tidy data principles in R","volume":"1","author":"Silge","year":"2016","journal-title":"J Open Source Softw"},{"key":"2026101108341593600_ooag163-B40","author":"Bouchet-Valat","year":"2023"},{"key":"2026101108341593600_ooag163-B41","author":"Chen","year":"2024"},{"key":"2026101108341593600_ooag163-B42","author":"Wilson","year":"2022"},{"key":"2026101108341593600_ooag163-B43","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v028.i05","article-title":"Building predictive models in R using the caret package","volume":"28","author":"Kuhn","year":"2008","journal-title":"J Stat Softw"},{"key":"2026101108341593600_ooag163-B44","author":"Harrell FE, Jr","year":"2024"},{"key":"2026101108341593600_ooag163-B45","author":"Zobolas"},{"key":"2026101108341593600_ooag163-B46","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1186\/1471-2105-12-77","article-title":"pROC: an open-source package for R and S + to analyze and compare ROC curves","volume":"12","author":"Robin","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2026101108341593600_ooag163-B47","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1093\/bioinformatics\/btw570","article-title":"Precrec: fast and accurate precision\u2013recall and ROC curve calculations in R","volume":"33","author":"Saito","year":"2017","journal-title":"Bioinformatics"},{"key":"2026101108341593600_ooag163-B48","doi-asserted-by":"publisher","first-page":"1696","DOI":"10.1093\/jamia\/ocac109","article-title":"Identifying infected patients using semi-supervised and transfer learning","volume":"29","author":"Bashiri","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2026101108341593600_ooag163-B49","doi-asserted-by":"publisher","first-page":"1284","DOI":"10.1097\/01.CCM.0000260960.94300.DE","article-title":"Sepsis incidence and outcome: Contrasting the intensive care unit with the hospital ward","volume":"35","author":"Esteban","year":"2007","journal-title":"Crit Care Med"},{"key":"2026101108341593600_ooag163-B50","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1002\/jhm.2012","article-title":"The epidemiology of acute organ system dysfunction from severe sepsis outside of the intensive care unit","volume":"8","author":"Rohde","year":"2013","journal-title":"J Hosp Med"},{"key":"2026101108341593600_ooag163-B51","doi-asserted-by":"publisher","first-page":"104335","DOI":"10.1016\/j.jbi.2023.104335","article-title":"Trends and opportunities in computable clinical phenotyping: a scoping review","volume":"140","author":"He","year":"2023","journal-title":"J Biomed Inform"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/5\/ooag163\/71839664\/ooag163.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/5\/ooag163\/71839664\/ooag163.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,11]],"date-time":"2026-10-11T12:34:24Z","timestamp":1791722064000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag163\/8907421"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":51,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,9,9]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag163","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,10]]},"published":{"date-parts":[[2026,9,9]]},"article-number":"ooag163"}}