{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T08:28:50Z","timestamp":1765268930092,"version":"3.37.3"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,8,30]],"date-time":"2023-08-30T00:00:00Z","timestamp":1693353600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR003528","R01LM013337"],"award-info":[{"award-number":["U01TR003528","R01LM013337"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","award":["5T32LM01220304"],"award-info":[{"award-number":["5T32LM01220304"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Institute of Child Health & Human Development","award":["R01HD105939"],"award-info":[{"award-number":["R01HD105939"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,12,22]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Bacterial infections (BIs) are common, costly, and potentially life-threatening in critically ill patients. Patients with suspected BIs may require empiric multidrug antibiotic regimens and therefore potentially be exposed to prolonged and unnecessary antibiotics. We previously developed a BI risk model to augment practices and help shorten the duration of unnecessary antibiotics to improve patient outcomes. Here, we have performed a transportability assessment of this BI risk model in 2 tertiary intensive care unit (ICU) settings and a community ICU setting. We additionally explored how simple multisite learning techniques impacted model transportability.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Patients suspected of having a community-acquired BI were identified in 3 datasets: Medical Information Mart for Intensive Care III (MIMIC), Northwestern Medicine Tertiary (NM-T) ICUs, and NM \u201ccommunity-based\u201d ICUs. ICU encounters from MIMIC and NM-T datasets were split into 70\/30 train and test sets. Models developed on training data were evaluated against the NM-T and MIMIC test sets, as well as NM community validation data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>During internal validations, models achieved AUROCs of 0.78 (MIMIC) and 0.81 (NM-T) and were well calibrated. In the external community ICU validation, the NM-T model had robust transportability (AUROC 0.81) while the MIMIC model transported less favorably (AUROC 0.74), likely due to case-mix differences. Multisite learning provided no significant discrimination benefit in internal validation studies but offered more stability during transport across all evaluation datasets.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>These results suggest that our BI risk models maintain predictive utility when transported to external cohorts.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Our findings highlight the importance of performing external model validation on myriad clinically relevant populations prior to implementation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad174","type":"journal-article","created":{"date-parts":[[2023,8,30]],"date-time":"2023-08-30T22:17:00Z","timestamp":1693433820000},"page":"98-108","source":"Crossref","is-referenced-by-count":6,"title":["Transportability of bacterial infection prediction models for critically ill patients"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5732-735X","authenticated-orcid":false,"given":"Garrett","family":"Eickelberg","sequence":"first","affiliation":[{"name":"Department of Preventive Medicine (Health & Biomedical Informatics), Feinberg School of Medicine , Chicago, IL 60611, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7434-6747","authenticated-orcid":false,"given":"Lazaro Nelson","family":"Sanchez-Pinto","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine (Health & Biomedical Informatics), Feinberg School of Medicine , Chicago, IL 60611, United States"},{"name":"Departments of Pediatrics (Critical Care) , Chicago, IL 60611, United States"}]},{"given":"Adrienne Sarah","family":"Kline","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine (Health & Biomedical Informatics), Feinberg School of Medicine , Chicago, IL 60611, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0195-7456","authenticated-orcid":false,"given":"Yuan","family":"Luo","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine (Health & Biomedical Informatics), Feinberg School of Medicine , Chicago, IL 60611, United States"}]}],"member":"286","published-online":{"date-parts":[[2023,8,30]]},"reference":[{"issue":"21","key":"2023122220311033500_ocad174-B1","doi-asserted-by":"crossref","first-page":"2323","DOI":"10.1001\/jama.2009.1754","article-title":"International study of the prevalence and outcomes of infection in intensive care units","volume":"302","author":"Vincent","year":"2009","journal-title":"JAMA"},{"issue":"15","key":"2023122220311033500_ocad174-B2","doi-asserted-by":"crossref","first-page":"1478","DOI":"10.1001\/jama.2020.2717","article-title":"Prevalence and outcomes of infection among patients in intensive care units in 2017","volume":"323","author":"Vincent","year":"2020","journal-title":"JAMA"},{"issue":"1","key":"2023122220311033500_ocad174-B3","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1053\/j.scrs.2017.09.008","article-title":"The risk of prescribing antibiotics \u201cjust-in-case\u201d there is infection","volume":"29","author":"Goff","year":"2018","journal-title":"Semin Colon Rectal Surg"},{"issue":"11","key":"2023122220311033500_ocad174-B4","doi-asserted-by":"crossref","first-page":"e1063","DOI":"10.1097\/CCM.0000000000005337","article-title":"Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021","volume":"49","author":"Evans","year":"2021","journal-title":"Crit Care Med"},{"issue":"5","key":"2023122220311033500_ocad174-B5","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1513\/AnnalsATS.202003-188ST","article-title":"Antibiotic stewardship in the intensive care unit. An Official American Thoracic Society Workshop Report in collaboration with the AACN, CHEST, CDC, and SCCM","volume":"17","author":"Wunderink","year":"2020","journal-title":"Ann Am Thorac Soc"},{"volume-title":"Antibiotic Use","year":"2019","author":"Core Elements of Hospital Antibiotic Stewardship Programs","key":"2023122220311033500_ocad174-B6"},{"issue":"12","key":"2023122220311033500_ocad174-B7","doi-asserted-by":"crossref","first-page":"647","DOI":"10.1177\/0885066618762747","article-title":"Antibiotic use in the intensive care unit: optimization and de-escalation","volume":"33","author":"Campion","year":"2018","journal-title":"J Intensive Care Med"},{"issue":"5","key":"2023122220311033500_ocad174-B8","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1186\/s13054-014-0480-6","article-title":"Antibiotic stewardship in the intensive care unit","volume":"18","author":"Luyt","year":"2014","journal-title":"Crit Care"},{"issue":"9","key":"2023122220311033500_ocad174-B9","doi-asserted-by":"crossref","first-page":"1308","DOI":"10.1001\/jamainternmed.2017.1938","article-title":"Association of adverse events with antibiotic use in hospitalized patients","volume":"177","author":"Tamma","year":"2017","journal-title":"JAMA Intern Med"},{"issue":"2","key":"2023122220311033500_ocad174-B10","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1089\/sur.2009.047","article-title":"Critical analysis of empiric antibiotic utilization: establishing benchmarks","volume":"11","author":"Claridge","year":"2010","journal-title":"Surg Infect (Larchmt)"},{"key":"2023122220311033500_ocad174-B11","first-page":"1543","article-title":"Antibiotics and the human gut microbiome: dysbioses and accumulation of resistances","volume":"6","author":"Francino","year":"2015","journal-title":"Front Microbiol"},{"issue":"12","key":"2023122220311033500_ocad174-B12","doi-asserted-by":"crossref","first-page":"2527","DOI":"10.1097\/CCM.0000000000001294","article-title":"A multicenter evaluation of prolonged empiric antibiotic therapy in adult ICUs in the United States","volume":"43","author":"Thomas","year":"2015","journal-title":"Crit Care Med"},{"issue":"6","key":"2023122220311033500_ocad174-B13","doi-asserted-by":"crossref","first-page":"839","DOI":"10.1097\/CCM.0000000000003143","article-title":"Society of critical care medicine presidential address \u2212 47th Annual Congress, February 2018, San Antonio, Texas","volume":"46","author":"Zimmerman","year":"2018","journal-title":"Crit Care Med"},{"key":"2023122220311033500_ocad174-B14","doi-asserted-by":"crossref","first-page":"103540","DOI":"10.1016\/j.jbi.2020.103540","article-title":"Predictive modeling of bacterial infections and antibiotic therapy needs in critically ill adults","volume":"109","author":"Eickelberg","year":"2020","journal-title":"J Biomed Inform"},{"issue":"1","key":"2023122220311033500_ocad174-B15","doi-asserted-by":"crossref","first-page":"W1","DOI":"10.7326\/M14-0698","article-title":"Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration","volume":"162","author":"Moons","year":"2015","journal-title":"Ann Intern Med"},{"issue":"7","key":"2023122220311033500_ocad174-B16","doi-asserted-by":"crossref","first-page":"1411","DOI":"10.1093\/jamia\/ocab018","article-title":"Validation of an internationally derived patient severity phenotype to support COVID-19 analytics from electronic health record data","volume":"28","author":"Klann","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2023122220311033500_ocad174-B17","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1186\/1471-2288-14-40","article-title":"External validation of multivariable prediction models: a systematic review of methodological conduct and reporting","volume":"14","author":"Collins","year":"2014","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"2023122220311033500_ocad174-B18","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1093\/ckj\/sfaa188","article-title":"External validation of prognostic models: what, why, how, when and where?","volume":"14","author":"Ramspek","year":"2021","journal-title":"Clin Kidney J"},{"issue":"8","key":"2023122220311033500_ocad174-B19","doi-asserted-by":"crossref","first-page":"e209271","DOI":"10.1001\/jamanetworkopen.2020.9271","article-title":"Derivation and validation of novel phenotypes of multiple organ dysfunction syndrome in critically ill children","volume":"3","author":"Sanchez-Pinto","year":"2020","journal-title":"JAMA Netw Open"},{"key":"2023122220311033500_ocad174-B20","doi-asserted-by":"crossref","first-page":"i3140","DOI":"10.1136\/bmj.i3140","article-title":"External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges","volume":"353","author":"Riley","year":"2016","journal-title":"BMJ"},{"issue":"3","key":"2023122220311033500_ocad174-B21","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.jclinepi.2014.06.018","article-title":"A new framework to enhance the interpretation of external validation studies of clinical prediction models","volume":"68","author":"Debray","year":"2015","journal-title":"J Clin Epidemiol"},{"issue":"7","key":"2023122220311033500_ocad174-B22","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1001\/jama.2021.24935","article-title":"Proactive vs reactive machine learning in health care: lessons from the COVID-19 pandemic","volume":"327","author":"Luo","year":"2022","journal-title":"JAMA"},{"issue":"1","key":"2023122220311033500_ocad174-B23","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1186\/s12916-019-1466-7","article-title":"Calibration: the Achilles heel of predictive analytics","volume":"17","author":"Van Calster","year":"2019","journal-title":"BMC Med"},{"issue":"8","key":"2023122220311033500_ocad174-B24","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1093\/aje\/kwq223","article-title":"External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients","volume":"172","author":"Vergouwe","year":"2010","journal-title":"Am J Epidemiol"},{"issue":"8","key":"2023122220311033500_ocad174-B25","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern Med"},{"volume-title":"The MIMIC-III Clinical Database","year":"2016","author":"Johnson","key":"2023122220311033500_ocad174-B26"},{"issue":"1","key":"2023122220311033500_ocad174-B27","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"issue":"2","key":"2023122220311033500_ocad174-B28","doi-asserted-by":"crossref","first-page":"ooac026","DOI":"10.1093\/jamiaopen\/ooac026","article-title":"Development and validation of MicrobEx: an open-source package for microbiology culture concept extraction","volume":"5","author":"Eickelberg","year":"2022","journal-title":"JAMIA Open"},{"volume-title":"ATC Classification Index with DDDs","year":"2019","author":"Methodology WCCfDS","key":"2023122220311033500_ocad174-B29"},{"key":"2023122220311033500_ocad174-B30","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"issue":"1","key":"2023122220311033500_ocad174-B31","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach Learn"},{"key":"2023122220311033500_ocad174-B32","first-page":"3","article-title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","volume":"10","author":"Platt","year":"2000","journal-title":"Adv Large Margin Classif"},{"first-page":"625","year":"2005","author":"Niculescu-Mizil","key":"2023122220311033500_ocad174-B33"},{"issue":"1","key":"2023122220311033500_ocad174-B34","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1186\/s12911-022-01879-6","article-title":"Learning patient-level prediction models across multiple healthcare databases: evaluation of ensembles for increasing model transportability","volume":"22","author":"Reps","year":"2022","journal-title":"BMC Med Inform Decis Mak"},{"issue":"3","key":"2023122220311033500_ocad174-B35","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"issue":"11","key":"2023122220311033500_ocad174-B36","doi-asserted-by":"crossref","first-page":"1389","DOI":"10.1109\/LSP.2014.2337313","article-title":"Fast implementation of DeLong\u2019s algorithm for comparing the areas under correlated receiver operating characteristic curves","volume":"21","author":"Sun","year":"2014","journal-title":"IEEE Signal Process Lett"},{"issue":"4","key":"2023122220311033500_ocad174-B37","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1093\/jamia\/ocz228","article-title":"A tutorial on calibration measurements and calibration models for clinical prediction models","volume":"27","author":"Huang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2023122220311033500_ocad174-B38","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.jclinepi.2015.12.005","article-title":"A calibration hierarchy for risk models was defined: from utopia to empirical data","volume":"74","author":"Van Calster","year":"2016","journal-title":"J Clin Epidemiol"},{"issue":"14","key":"2023122220311033500_ocad174-B39","doi-asserted-by":"crossref","first-page":"1429","DOI":"10.1001\/jama.2018.1536","article-title":"The proposal to lower P value thresholds to.005","volume":"319","author":"Ioannidis","year":"2018","journal-title":"JAMA"},{"first-page":"1171","year":"2017","author":"Zafar","key":"2023122220311033500_ocad174-B40"},{"first-page":"3323","year":"2016","author":"Hardt","key":"2023122220311033500_ocad174-B41"},{"issue":"2","key":"2023122220311033500_ocad174-B42","doi-asserted-by":"crossref","first-page":"e0148820","DOI":"10.1371\/journal.pone.0148820","article-title":"Assessing discriminative performance at external validation of clinical prediction models","volume":"11","author":"Nieboer","year":"2016","journal-title":"PLoS One"},{"issue":"Suppl 1","key":"2023122220311033500_ocad174-B43","first-page":"S1","article-title":"Guidelines for antibiotic prescription in intensive care unit","volume":"23","author":"Khilnani","year":"2019","journal-title":"Indian J Crit Care Med"},{"issue":"5","key":"2023122220311033500_ocad174-B44","doi-asserted-by":"crossref","first-page":"1496","DOI":"10.1378\/chest.117.5.1496","article-title":"Rational empiric antibiotic prescription in the ICU","volume":"117","author":"Singh","year":"2000","journal-title":"Chest"},{"issue":"2","key":"2023122220311033500_ocad174-B45","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1097\/CCM.0b013e31827e83af","article-title":"Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012","volume":"41","author":"Dellinger","year":"2013","journal-title":"Crit Care Med"},{"key":"2023122220311033500_ocad174-B46","first-page":"196","article-title":"Assessment of data quality variability across two EHR systems through a case study of post-surgical complications","volume":"2022","author":"Fu","year":"2022","journal-title":"AMIA Annu Symp Proc"},{"key":"2023122220311033500_ocad174-B47","first-page":"1109","article-title":"Developing predictive models using electronic medical records: challenges and pitfalls","volume":"2013","author":"Paxton","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"issue":"4","key":"2023122220311033500_ocad174-B48","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1016\/j.annemergmed.2020.11.007","article-title":"Predicting progression to septic shock in the emergency department using an externally generalizable machine-learning algorithm","volume":"77","author":"Wardi","year":"2021","journal-title":"Ann Emerg Med"},{"issue":"4","key":"2023122220311033500_ocad174-B49","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","article-title":"An interpretable machine learning model for accurate prediction of sepsis in the ICU","volume":"46","author":"Nemati","year":"2018","journal-title":"Crit Care Med"},{"issue":"Suppl 5","key":"2023122220311033500_ocad174-B50","first-page":"1","article-title":"Unsupervised phenotyping of sepsis using nonnegative matrix factorization of temporal trends from a multivariate panel of physiological measurements","volume":"21","author":"Ding","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"year":"2021","author":"Shin","key":"2023122220311033500_ocad174-B51"},{"issue":"Suppl 2","key":"2023122220311033500_ocad174-B52","first-page":"1","article-title":"Comparison between machine learning methods for mortality prediction for sepsis patients with different social determinants","volume":"22","author":"Wang","year":"2022","journal-title":"BMC Med Inform Decis Mak"},{"year":"2019","author":"Corey","key":"2023122220311033500_ocad174-B53"},{"issue":"6","key":"2023122220311033500_ocad174-B54","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1016\/S1473-3099(18)30273-1","article-title":"Grading antimicrobial susceptibility data quality: room for improvement","volume":"18","author":"Ashley","year":"2018","journal-title":"Lancet Infect Dis"},{"issue":"1","key":"2023122220311033500_ocad174-B55","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1186\/s12916-019-1301-1","article-title":"Microbiology Investigation Criteria for Reporting Objectively (MICRO): a framework for the reporting and interpretation of clinical microbiology data","volume":"17","author":"Turner","year":"2019","journal-title":"BMC Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/1\/98\/54762220\/ocad174.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/1\/98\/54762220\/ocad174.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,22]],"date-time":"2023-12-22T20:32:18Z","timestamp":1703277138000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/1\/98\/7255953"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,30]]},"references-count":55,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,8,30]]},"published-print":{"date-parts":[[2023,12,22]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad174","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"type":"print","value":"1067-5027"},{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2024,1,1]]},"published":{"date-parts":[[2023,8,30]]}}}