{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T09:43:42Z","timestamp":1781343822748,"version":"3.54.1"},"reference-count":15,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T00:00:00Z","timestamp":1753142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000133","name":"Agency for Healthcare Research and Quality","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000133","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["5R03HS029417-02"],"award-info":[{"award-number":["5R03HS029417-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UL1TR002378"],"award-info":[{"award-number":["UL1TR002378"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["KL2TR002381"],"award-info":[{"award-number":["KL2TR002381"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To conduct an independent external validation of an implemented vendor-developed emergency department (ED) pediatric sepsis predictive model.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We performed a retrospective cross-sectional study within 2 ED sites of a large pediatric health system between January 1, 2021 and April 1, 2024. A nurse-facing interruptive alert appeared when the model score exceeded the threshold, triggering clinicians to call a sepsis huddle. We compared model predictive performance with vendor-reported performance using definitions that accounted for model threshold and alert timing in clinical practice. Care processes and patient outcome measures included time to first antibiotics, time to first fluid bolus, 30-day mortality, ED to ICU admission rate, and ICU free days.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The pre-intervention cohort consisted of 268\u00a0102 ED visits with 741 (0.28%) sepsis cases. The post-intervention cohort consisted of 331\u00a0061 ED visits with 1114 (0.34%) sepsis cases. Model predictive performance dropped from vendor-reported performance. Mean time to first antibiotic decreased from 112 to 102 minutes (P\u2009=\u2009.05, 95% confidence interval of difference, \u221219.1 to 0.1) and time to first bolus decreased by 16.7 minutes (P\u2009=\u2009.03, 95% confidence interval difference, \u221231.8 to \u22121.5) after the intervention. Decreases in 30-day mortality (6% [45\/741] to 4% [52\/1114]); ED to ICU admissions (87% [646\/741] to 84% [941\/1114]), and ICU free days (6 to 5) after the intervention did not meet statistical significance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Implementing the model led to significant reductions in time to fluid bolus and borderline decreases in time to antibiotics, with non-significant changes in mortality and ICU metrics. When implementing an externally developed model, local workflows, documentation patterns, and patient populations make it challenging to generalize published or reported model performance metrics to real world performance.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>When tailoring a vendor-developed pediatric ED sepsis model for real-world usage, predictive performance differed substantially. Post-implementation we found improvements in care process measures, suggesting such models may benefit sepsis care when adapted for specific clinical workflows.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf105","type":"journal-article","created":{"date-parts":[[2025,7,19]],"date-time":"2025-07-19T11:56:15Z","timestamp":1752926175000},"page":"1542-1551","source":"Crossref","is-referenced-by-count":3,"title":["Early clinical evaluation of a vendor developed pediatric artificial intelligence sepsis model in the emergency department"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2109-5769","authenticated-orcid":false,"given":"Swaminathan","family":"Kandaswamy","sequence":"first","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3756-8575","authenticated-orcid":false,"given":"Evan W","family":"Orenstein","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naveen","family":"Muthu","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"McCarter","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikolay","family":"Braykov","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan M","family":"Beus","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edwin","family":"Ray","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tal","family":"Senior","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sara P","family":"Brown","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebekah","family":"Carter","sequence":"additional","affiliation":[{"name":"Emergency Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"MaryBeth","family":"Gleeson","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hannah","family":"Thummel","sequence":"additional","affiliation":[{"name":"Children\u2019s Healthcare of Atlanta Clinical Process Improvement, , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Cheng","sequence":"additional","affiliation":[{"name":"Emergency Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thuy","family":"Bui","sequence":"additional","affiliation":[{"name":"Emergency Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Reena","family":"Blanco","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Emergency Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kiran","family":"Hebbar","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Critical Care Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Fortenberry","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Critical Care Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Srikant B","family":"Iyer","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Emergency Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0346-5818","authenticated-orcid":false,"given":"Mark V","family":"Mai","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]},{"name":"Critical Care Medicine, Children\u2019s Healthcare of Atlanta , Atlanta, GA 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"key":"2025092208422027000_ocaf105-B1","doi-asserted-by":"crossref","first-page":"E916","DOI":"10.1097\/CCM.0000000000004505","article-title":"Evaluating pediatric sepsis definitions designed for electronic health record extraction and multicenter quality improvement","volume":"48","author":"Scott","year":"2020","journal-title":"Crit Care Med"},{"key":"2025092208422027000_ocaf105-B2","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/S2352-4642(20)30341-2","article-title":"Hospital outcomes for children with severe sepsis in the USA by race or ethnicity and insurance status: a population-based, retrospective cohort study","volume":"5","author":"Mitchell","year":"2021","journal-title":"Lancet Child Adolesc Health"},{"key":"2025092208422027000_ocaf105-B3","doi-asserted-by":"crossref","first-page":"986","DOI":"10.1001\/jamapediatrics.2019.2570","article-title":"Cost of pediatric severe sepsis hospitalizations","volume":"173","author":"Carlton","year":"2019","journal-title":"JAMA Pediatr"},{"key":"2025092208422027000_ocaf105-B4","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/s00134-019-05872-y","article-title":"Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy","volume":"46","author":"Fleuren","year":"2020","journal-title":"Intensive Care Med"},{"key":"2025092208422027000_ocaf105-B5","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External Validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern Med"},{"key":"2025092208422027000_ocaf105-B6","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1097\/CCM.0000000000005267","article-title":"Improving timeliness of antibiotic administration using a provider and pharmacist facing sepsis early warning system in the emergency department setting: a randomized controlled quality improvement initiative\u2217","volume":"50","author":"Tarabichi","year":"2022","journal-title":"Crit Care Med"},{"key":"2025092208422027000_ocaf105-B7","doi-asserted-by":"crossref","first-page":"803","DOI":"10.1055\/s-0038-1675211","article-title":"Automating a manual sepsis screening tool in a pediatric emergency department","volume":"9","author":"Lloyd","year":"2018","journal-title":"Appl Clin Inform"},{"key":"2025092208422027000_ocaf105-B8","author":"Clark","year":"2018"},{"key":"2025092208422027000_ocaf105-B9"},{"key":"2025092208422027000_ocaf105-B10","doi-asserted-by":"crossref","first-page":"e20201434","DOI":"10.1542\/peds.2020-1434","article-title":"Development of a quality improvement learning collaborative to improve pediatric sepsis outcomes","volume":"147","author":"Larsen","year":"2021","journal-title":"Pediatrics"},{"key":"2025092208422027000_ocaf105-B11","author":"Singh","year":"2024"},{"key":"2025092208422027000_ocaf105-B12","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1097\/PCC.0000000000000858","article-title":"Protocolized treatment is associated with decreased organ dysfunction in pediatric severe sepsis","volume":"17","author":"Balamuth","year":"2016","journal-title":"Pediatr Crit Care Med"},{"key":"2025092208422027000_ocaf105-B13","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1016\/j.annemergmed.2017.03.019","article-title":"Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign-based electronic alert and bedside clinician identification","volume":"70","author":"Balamuth","year":"2017","journal-title":"Ann Emerg Med"},{"key":"2025092208422027000_ocaf105-B14","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1542\/hpeds.2022-006964","article-title":"Sepsis prediction in hospitalized children: model development and validation","volume":"13","author":"Stephen","year":"2023","journal-title":"Hosp Pediatr"},{"key":"2025092208422027000_ocaf105-B15","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1001\/jama.2024.0179","article-title":"International Consensus Criteria for pediatric sepsis and septic shock","volume":"331","author":"Schlapbach","year":"2024","journal-title":"JAMA"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/10\/1542\/63824376\/ocaf105.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/10\/1542\/63824376\/ocaf105.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,22]],"date-time":"2025-09-22T12:42:31Z","timestamp":1758544951000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/10\/1542\/8210593"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":15,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,7,22]]},"published-print":{"date-parts":[[2025,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf105","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,10]]},"published":{"date-parts":[[2025,7,22]]}}}