{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T10:30:36Z","timestamp":1778495436776,"version":"3.51.4"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T00:00:00Z","timestamp":1753142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000133","name":"Agency for Healthcare Research and Quality","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000133","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["5R03HS029417-02"],"award-info":[{"award-number":["5R03HS029417-02"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["UL1TR002378"],"award-info":[{"award-number":["UL1TR002378"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["KL2TR002381"],"award-info":[{"award-number":["KL2TR002381"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To assess the influence of an implemented artificial intelligence model predicting pediatric sepsis (defined by IPSO\u2014Improving Pediatric Sepsis Outcomes collaborative) in the emergency department (ED) on human performance measures.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Two ED sites within a large pediatric health system in the Southeastern United States between January 1, 2021 and April 1, 2024. We interviewed ED providers and nurses within 72 hours of caring for a patient identified as potentially having sepsis by the predictive model. Thematic analysis of qualitative data was combined with electronic health record queries to assess measures of human performance, including situation awareness, explainability, human-computer agreement, workload, trust, automation bias, and relationship between staff and patients.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We interviewed 40 clinicians. Participants found that the sepsis alert improved situation awareness, leading to changes in patient care management, resource allocation, and\/or monitoring. Participants reported an average trust in the model-based alert of 3.8\/5. Only 28% (555\/1977) of sepsis huddles were done without alert firing, suggesting some automation bias. Treatment with antibiotics for IPSO sepsis cases was similar pre- and post-intervention without a huddle (9.3% vs 10.5%), though treatment doubled with huddle intervention (22.7%). NASA Task Load Index increased from 43 to 57 post-intervention. There was no report of adverse relationships with patients post-intervention.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Human performance appeared to be generally positive with improved situation awareness and satisfaction with the alert-driven huddle. However, there was some evidence of automation bias and a slight increase in workload with the intervention<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>This study demonstrates the feasibility of evaluating multiple dimensions of human performance using a mixed methods approach for an AI model implemented in clinical practice. Future studies should aim to reduce the measurement burden of human performance metrics associated with AI implementation in acute care settings and assess the correlation between human performance measures and clinical outcomes.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf106","type":"journal-article","created":{"date-parts":[[2025,7,19]],"date-time":"2025-07-19T11:56:20Z","timestamp":1752926180000},"page":"1552-1561","source":"Crossref","is-referenced-by-count":2,"title":["Human performance evaluation of a pediatric artificial intelligence sepsis model"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2109-5769","authenticated-orcid":false,"given":"Swaminathan","family":"Kandaswamy","sequence":"first","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA, 30307,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naveen","family":"Muthu","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA, 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nikolay","family":"Braykov","sequence":"additional","affiliation":[{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rebekah","family":"Carter","sequence":"additional","affiliation":[{"name":"Emergency Medicine, Children's Healthcare of Atlanta, Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Reena","family":"Blanco","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA, 30307,","place":["United States"]},{"name":"Emergency Medicine, Children's Healthcare of Atlanta, Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thuy","family":"Bui","sequence":"additional","affiliation":[{"name":"Emergency Medicine, Children's Healthcare of Atlanta, Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3756-8575","authenticated-orcid":false,"given":"Evan","family":"Orenstein","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA, 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0346-5818","authenticated-orcid":false,"given":"Mark","family":"Mai","sequence":"additional","affiliation":[{"name":"Pediatrics, Emory University School of Medicine , Atlanta, GA, 30307,","place":["United States"]},{"name":"Information Systems & Technology, Children\u2019s Healthcare of Atlanta , Atlanta, GA, 30329,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"key":"2025092208421462200_ocaf106-B1","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.apergo.2018.06.008","article-title":"Human performance measures for the evaluation of process control human-system interfaces in high-fidelity simulations","volume":"73","author":"Xu","year":"2018","journal-title":"Appl Ergon"},{"key":"2025092208421462200_ocaf106-B2","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1136\/bmjqs-2020-012538","article-title":"SEIPS 101 and seven simple SEIPS tools","volume":"30","author":"Holden","year":"2021","journal-title":"BMJ Qual Saf"},{"key":"2025092208421462200_ocaf106-B3","doi-asserted-by":"crossref","first-page":"e8775","DOI":"10.2196\/jmir.8775","article-title":"Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies","volume":"19","author":"Greenhalgh","year":"2017","journal-title":"J Med Internet Res"},{"key":"2025092208421462200_ocaf106-B4","doi-asserted-by":"crossref","first-page":"e28639","DOI":"10.2196\/28639","article-title":"Human factors and technological characteristics influencing the interaction of medical professionals with artificial intelligence\u2013enabled clinical decision support systems: literature review","volume":"9","author":"Knop","year":"2022","journal-title":"JMIR Hum Factors"},{"key":"2025092208421462200_ocaf106-B5","first-page":"36","article-title":"Barriers and facilitators to clinical decision support systems adoption: a systematic review","volume":"3","author":"Devaraj","year":"2014","journal-title":"J Bus Adm Res"},{"key":"2025092208421462200_ocaf106-B6","first-page":"4477","article-title":"Investigating the heart pump implant decision process: opportunities for decision support tools to help","volume":"2016","author":"Yang","year":"2016","journal-title":"ACM Trans Comput-Hum Interact Publ Assoc Comput Mach"},{"key":"2025092208421462200_ocaf106-B7","article-title":"A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy","author":"Beede","year":"2020","journal-title":"Conf Hum Factors Comput Syst\u2014Proc"},{"key":"2025092208421462200_ocaf106-B8","article-title":"Understanding design for automated image analysis in digital pathology","author":"Molin","year":"2016","journal-title":"ACM Int Conf Proceeding Ser"},{"key":"2025092208421462200_ocaf106-B9","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1016\/j.jacr.2017.12.029","article-title":"Why CAD failed in mammography","volume":"15","year":"2018","journal-title":"J Am Coll Radiol"},{"key":"2025092208421462200_ocaf106-B10","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1007\/s40257-020-00574-4","article-title":"The importance of incorporating human factors in the design and implementation of artificial intelligence for skin cancer diagnosis in the real world","volume":"22","author":"Felmingham","year":"2021","journal-title":"Am J Clin Dermatol"},{"key":"2025092208421462200_ocaf106-B11","doi-asserted-by":"crossref","first-page":"e15182","DOI":"10.2196\/15182","article-title":"Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study","volume":"8","author":"Sendak","year":"2020","journal-title":"JMIR Med Inform"},{"key":"2025092208421462200_ocaf106-B12","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/3351095.3372827","author":"Sendak","year":"2020"},{"key":"2025092208421462200_ocaf106-B13","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1038\/s41591-022-01772-9","article-title":"Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"28","author":"Vasey","year":"2022","journal-title":"Nat Med"},{"key":"2025092208421462200_ocaf106-B14","doi-asserted-by":"crossref","first-page":"e1063","DOI":"10.1097\/CCM.0000000000005337","article-title":"Surviving sepsis campaign: International Guidelines for Management of Sepsis and Septic Shock 2021","volume":"49","author":"Evans","year":"2021","journal-title":"Crit Care Med"},{"key":"2025092208421462200_ocaf106-B15","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/s00134-019-05872-y","article-title":"Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy","volume":"46","author":"Fleuren","year":"2020","journal-title":"Intensive Care Med"},{"key":"2025092208421462200_ocaf106-B16","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern Med"},{"key":"2025092208421462200_ocaf106-B17","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1097\/CCM.0000000000005267","article-title":"Improving timeliness of antibiotic administration using a provider and pharmacist facing sepsis early warning system in the emergency department setting: a randomized controlled quality improvement initiative\u2217","volume":"50","author":"Tarabichi","year":"2022","journal-title":"Crit Care Med"},{"key":"2025092208421462200_ocaf106-B18","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1136\/svn-2017-000101","article-title":"Artificial intelligence in healthcare: past, present and future","volume":"2","author":"Jiang","year":"2017","journal-title":"Stroke Vasc Neurol"},{"key":"2025092208421462200_ocaf106-B19","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1518\/001872097778543886","article-title":"Humans and automation: use, misuse, disuse, abuse","volume":"39","author":"Parasuraman","year":"1997","journal-title":"Hum Factors J Hum Factors Ergon Soc"},{"key":"2025092208421462200_ocaf106-B20","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1038\/s41591-021-01229-5","article-title":"DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence","volume":"27","author":"Vasey","year":"2021","journal-title":"Nat Med"},{"key":"2025092208421462200_ocaf106-B21","doi-asserted-by":"crossref","first-page":"594971","DOI":"10.3389\/fdgth.2021.594971","article-title":"Success factors of artificial intelligence implementation in healthcare","volume":"3","author":"Wolff","year":"2021","journal-title":"Front Digit Health"},{"key":"2025092208421462200_ocaf106-B22","doi-asserted-by":"crossref","first-page":"e18599","DOI":"10.2196\/18599","article-title":"Role of artificial intelligence in patient safety outcomes: systematic literature review","volume":"8","author":"Choudhury","year":"2020","journal-title":"JMIR Med Inform"},{"key":"2025092208421462200_ocaf106-B23","doi-asserted-by":"crossref","DOI":"10.1542\/peds.2020-1434","article-title":"Development of a quality improvement learning collaborative to improve pediatric sepsis outcomes","volume":"147","author":"Larsen","year":"2021","journal-title":"Pediatrics"},{"key":"2025092208421462200_ocaf106-B24","doi-asserted-by":"crossref","first-page":"100516","DOI":"10.1136\/bmjhci-2021-100516","article-title":"Eight human factors and ergonomics principles for healthcare artificial intelligence","volume":"29","author":"Sujan","year":"2022","journal-title":"BMJ Health Care Inf"},{"issue":"1","key":"2025092208421462200_ocaf106-B25","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1518\/001872095779049543","article-title":"Toward a theory of situation awareness in dynamic systems","volume":"37","author":"Endsley","journal-title":"Hum Factors"},{"key":"2025092208421462200_ocaf106-B26","author":"Hussain","year":"2021"},{"key":"2025092208421462200_ocaf106-B27","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"NASA-Task Load Index (NASA-TLX); 20 years later","volume":"50","author":"Hart","year":"2006","journal-title":"Proc Hum Factors Ergon Soc Annu Meet"},{"key":"2025092208421462200_ocaf106-B28","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1177\/0018720810376055","article-title":"Complacency and bias in human use of automation: an attentional integration","volume":"52","author":"Parasuraman","year":"2010","journal-title":"Hum Factors"},{"key":"2025092208421462200_ocaf106-B29","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2025092208421462200_ocaf106-B30","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1518\/hfes.46.1.50.30392","article-title":"Trust in automation: designing for appropriate reliance","volume":"46","author":"Lee","year":"2004","journal-title":"Hum Factors"},{"key":"2025092208421462200_ocaf106-B31","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s11577-017-0454-1","article-title":"How to construct a mixed methods research design","volume":"69","author":"Schoonenboom","year":"2017","journal-title":"Kolner Z Soziol Sozialpsychologie"},{"key":"2025092208421462200_ocaf106-B32","first-page":"1","article-title":"Measuring situation awareness in emergency settings: a systematic review of tools and outcomes","volume":"6","author":"Cooper","year":"2014","journal-title":"Open Access Emerg Med OAEM"},{"key":"2025092208421462200_ocaf106-B33","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1080\/10580530.2020.1849465","article-title":"explainable artificial intelligence: objectives, stakeholders, and future research opportunities","volume":"39","author":"Meske","year":"2022","journal-title":"Inf Syst Manag [Internet]"},{"key":"2025092208421462200_ocaf106-B34","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1055\/s-0042-1748856","article-title":"Clinical decision support stewardship: best practices and techniques to monitor and improve interruptive alerts","volume":"13","author":"Chaparro","year":"2022","journal-title":"Appl Clin Inform"},{"key":"2025092208421462200_ocaf106-B35","first-page":"500","article-title":"Alert fatigue and errors caused by technology: a scoping review and introduction to the flow of cognitive processing model","volume":"13","author":"Joseph","year":"2021","journal-title":"Knowl Manag E-Learn Int J"},{"key":"2025092208421462200_ocaf106-B36","doi-asserted-by":"crossref","first-page":"e62763","DOI":"10.2196\/62763","article-title":"Understanding \u201calert fatigue\u201d in primary care: qualitative systematic review of general practitioners attitudes and experiences of clinical alerts, prompts, and reminders","volume":"27","author":"Gani","year":"2025","journal-title":"J Med Internet Res"},{"key":"2025092208421462200_ocaf106-B37","first-page":"160","author":"Bussone","year":"2015"},{"key":"2025092208421462200_ocaf106-B38","doi-asserted-by":"crossref","first-page":"e46407","DOI":"10.2196\/46407","article-title":"Evaluating artificial intelligence in clinical settings\u2014let us not reinvent the wheel","volume":"26","author":"Cresswell","year":"2024","journal-title":"J Med Internet Res"},{"key":"2025092208421462200_ocaf106-B39","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1542\/hpeds.2022-006964","article-title":"Sepsis prediction in hospitalized children: model development and validation","volume":"13","author":"Stephen","year":"2023","journal-title":"Hosp Pediatr"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/10\/1552\/63824370\/ocaf106.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/10\/1552\/63824370\/ocaf106.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,22]],"date-time":"2025-09-22T12:42:22Z","timestamp":1758544942000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/10\/1552\/8210592"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":39,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,7,22]]},"published-print":{"date-parts":[[2025,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf106","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,10]]},"published":{"date-parts":[[2025,7,22]]}}}