{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T13:01:06Z","timestamp":1791464466200,"version":"4.3.4"},"reference-count":37,"publisher":"Georg Thieme Verlag KG","issue":"04","funder":[{"DOI":"10.13039\/100000009","name":"Foundation for the National Institutes of Health","doi-asserted-by":"crossref","award":["1R01LM013526-01A1"],"award-info":[{"award-number":["1R01LM013526-01A1"]}],"id":[{"id":"10.13039\/100000009","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Human-centered design (HCD) methods in machine learning generally focus on workflow, user interfaces, and data visualizations, but there is the potential to apply these methods to inform the model development and testing process.<\/jats:p><jats:p>This study aimed to demonstrate the potential of HCD methods to support the design and testing of machine learning models developed for clinical decision-making.<\/jats:p><jats:p>In preparing for formative user testing of clinician facing representations of a machine learning model for detecting sepsis in neonatal intensive care unit (NICU) patients, we discovered that interactive low fidelity mockups using real patient data revealed potential model anomalies. To further investigate these potential anomalies, we utilized the qualitative analysis of interviews with 31 NICU clinicians concerning their experience with neonatal sepsis. The review process was conducted by a multidisciplinary team with members having expertise in neonatology, informatics, data science, and human computer interaction (HCI). Anomalies identified via the mockups and interview analysis were further analyzed by inspections of patient charts and model features and code.<\/jats:p><jats:p>The HCD-facilitated review revealed anomalies in three categories: (1) feature inclusion and exclusion, (2) feature importance, and (3) model stability over time. Data entry errors in the electronic health record and their impact on model output were also noted. The review resulted in 41 changes to the model.<\/jats:p><jats:p>The discovery of over 41 opportunities to improve our prediction model was a serendipitous by-product of the HCD process. Our results suggest that HCD can be applied not only to model display design and measures of explainability, but to the development and evaluation of the model itself. This case report also demonstrates the need for a multidisciplinary team of clinicians, data scientists, and HCI experts in identifying and addressing issues involving machine learning model performance.<\/jats:p>","DOI":"10.1055\/a-2618-4470","type":"journal-article","created":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:18:58Z","timestamp":1760138338000},"page":"1332-1340","source":"Crossref","is-referenced-by-count":1,"title":["Refining a Machine Learning Model for Predicting Infant Sepsis: A Multidisciplinary Team Supported by Human-Centered Design Methods"],"prefix":"10.1055","volume":"16","author":[{"given":"Dean","family":"Karavite","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lusha","family":"Cao","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mary C.","family":"Harris","sequence":"additional","affiliation":[{"name":"Department of Neonatology, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States"},{"name":"Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Fidel","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lyle","family":"Ungar","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gerald","family":"Shaeffer","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Xiao","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick","family":"Brady","sequence":"additional","affiliation":[{"name":"Cincinnati Children's Hospital Medical Center, Department of Pediatrics, Perinatal Institute and James M. Anderson Center for Health Systems Excellence, Cincinnati, Ohio, United States"},{"name":"Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heather C.","family":"Kaplan","sequence":"additional","affiliation":[{"name":"Cincinnati Children's Hospital Medical Center, Department of Pediatrics, Perinatal Institute and James M. Anderson Center for Health Systems Excellence, Cincinnati, Ohio, United States"},{"name":"Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert W.","family":"Grundmeier","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Health Informatics, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States"},{"name":"Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"194","published-online":{"date-parts":[[2025,10,10]]},"reference":[{"issue":"06","key":"ref1","doi-asserted-by":"crossref","first-page":"1056","DOI":"10.1016\/j.jbi.2011.08.005","article-title":"TURF: toward a unified framework of EHR usability","volume":"44","author":"J Zhang","year":"2011","journal-title":"J Biomed Inform"},{"key":"ref2","first-page":"1085","volume-title":"Human-centered design of artificial intelligence","author":"G Margetis","year":"2021","edition":"1st ed."},{"key":"ref4","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1016\/j.future.2022.05.014","article-title":"A survey of human-in-the-loop for machine learning","volume":"135","author":"X Wu","year":"2022","journal-title":"Future Gener Comput Syst"},{"issue":"04","key":"ref5","doi-asserted-by":"crossref","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","article-title":"Human-in-the-loop machine learning: a state of the art","volume":"56","author":"E Mosqueira-Rey","year":"2023","journal-title":"Artif Intell Rev"},{"issue":"02","key":"ref6","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","article-title":"Interactive machine learning for health informatics: when do we need the human-in-the-loop?","volume":"3","author":"A Holzinger","year":"2016","journal-title":"Brain Inform"},{"issue":"09","key":"ref7","doi-asserted-by":"crossref","first-page":"4571","DOI":"10.1073\/pnas.1906831117","article-title":"Expert-augmented machine learning","volume":"117","author":"E D Gennatas","year":"2020","journal-title":"Proc Natl Acad Sci U S A"},{"key":"ref8","doi-asserted-by":"crossref","first-page":"37854","DOI":"10.1038\/srep37854","article-title":"MediBoost: a patient stratification tool for interpretable decision making in the era of precision medicine","volume":"6","author":"G Valdes","year":"2016","journal-title":"Sci Rep"},{"issue":"01","key":"ref9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/tropej\/fmu079","article-title":"Challenges in the diagnosis and management of neonatal sepsis","volume":"61","author":"A Zea-Vera","year":"2015","journal-title":"J Trop Pediatr"},{"key":"ref10","doi-asserted-by":"crossref","first-page":"1725","DOI":"10.1016\/S0140-6736(16)31575-6","article-title":"Global, regional, national, and selected subnational levels of stillbirths, neonatal, infant, and under-5 mortality, 1980-2015: a systematic analysis for the Global Burden of Disease Study 2015","volume":"388","author":"H Wang","year":"2016","journal-title":"Lancet"},{"issue":"05","key":"ref11","doi-asserted-by":"crossref","first-page":"2335","DOI":"10.1007\/s00431-023-04911-7","article-title":"Global, regional, and national burden of neonatal sepsis and other neonatal infections, 1990-2019: findings from the Global Burden of Disease Study 2019","volume":"182","author":"J Li","year":"2023","journal-title":"Eur J Pediatr"},{"issue":"03","key":"ref12","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1055\/s-0040-1710525","article-title":"A review of predictive analytics solutions for sepsis patients","volume":"11","author":"A K Teng","year":"2020","journal-title":"Appl Clin Inform"},{"issue":"05","key":"ref13","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1542\/peds.2010-2217","article-title":"Early onset neonatal sepsis: the burden of group B Streptococcal and E. coli disease continues","volume":"127","author":"B J Stoll","year":"2011","journal-title":"Pediatrics"},{"key":"ref14","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1097\/INF.0b013e31819587a9","article-title":"Neonatal sepsis: a major global public health challenge","volume":"28","author":"S A Qazi","year":"2009","journal-title":"Pediatr Infect Dis J"},{"issue":"02","key":"ref15","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1097\/MOP.0b013e3283504df3","article-title":"New technologies for the rapid diagnosis of neonatal sepsis","volume":"24","author":"L Srinivasan","year":"2012","journal-title":"Curr Opin Pediatr"},{"issue":"02","key":"ref16","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1097\/MOP.0000000000000315","article-title":"Defining neonatal sepsis","volume":"28","author":"J L Wynn","year":"2016","journal-title":"Curr Opin Pediatr"},{"issue":"01","key":"ref17","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1055\/a-1990-3037","article-title":"SEPRES: intensive care unit clinical data integration system to predict sepsis","volume":"14","author":"Q Chen","year":"2023","journal-title":"Appl Clin Inform"},{"issue":"02","key":"ref18","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1055\/s-0040-1705107","article-title":"Design, Implementation, and Validation of a Pediatric ICU sepsis prediction tool as clinical decision support","volume":"11","author":"M Dewan","year":"2020","journal-title":"Appl Clin Inform"},{"issue":"02","key":"ref19","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1055\/s-0039-1688478","article-title":"Diffusing an innovation: clinician perceptions of continuous predictive analytics monitoring in intensive care","volume":"10","author":"R R Kitzmiller","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"11","key":"ref20","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1097\/CCM.0000000000003803","article-title":"Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock","volume":"47","author":"J C Ginestra","year":"2019","journal-title":"Crit Care Med"},{"issue":"02","key":"ref21","doi-asserted-by":"crossref","first-page":"e0212665","DOI":"10.1371\/journal.pone.0212665","article-title":"Machine learning models for early sepsis recognition in the neonatal intensive care unit using readily available electronic health record data","volume":"14","author":"A J Masino","year":"2019","journal-title":"PLoS One"},{"issue":"02","key":"ref23","doi-asserted-by":"crossref","first-page":"e57","DOI":"10.1055\/s-0042-1749318","article-title":"Using a sociotechnical model to understand challenges with sepsis recognition among critically ill infants","volume":"06","author":"D J Karavite","year":"2022","journal-title":"ACI Open"},{"key":"ref24","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.jbi.2018.01.001","article-title":"Ten factors to consider when developing usability scenarios and tasks for health information technology","volume":"78","author":"A L Russ","year":"2018","journal-title":"J Biomed Inform"},{"issue":"01","key":"ref25","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1186\/s12911-020-01276-x","article-title":"A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare","volume":"20","author":"A J Barda","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"ref27","doi-asserted-by":"crossref","first-page":"S137","DOI":"10.7326\/M19-0872","article-title":"Reporting and implementing interventions involving machine learning and artificial intelligence","volume":"172","author":"D W Bates","year":"2020","journal-title":"Ann Intern Med"},{"issue":"6464","key":"ref28","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Z Obermeyer","year":"2019","journal-title":"Science"},{"key":"ref29","doi-asserted-by":"crossref","first-page":"102423","DOI":"10.1016\/j.artmed.2022.102423","article-title":"A manifesto on explainability for artificial intelligence in medicine","volume":"133","author":"C Combi","year":"2022","journal-title":"Artif Intell Med"},{"key":"ref30","doi-asserted-by":"crossref","first-page":"103473","DOI":"10.1016\/j.artint.2021.103473","article-title":"What do we want from explainable artificial intelligence (XAI)?\u2013A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research","volume":"296","author":"M Langer","year":"2021","journal-title":"Artif Intell"},{"issue":"06","key":"ref32","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1080\/10447318.2020.1741118","article-title":"Human-centered artificial intelligence: reliable, safe & trustworthy","volume":"36","author":"B Shneiderman","year":"2020","journal-title":"Int J Hum Comput Interact"},{"key":"ref33","doi-asserted-by":"crossref","first-page":"1187299","DOI":"10.3389\/fcomp.2023.1187299","article-title":"Human-centered design and evaluation of AI-empowered clinical decision support systems: a systematic review","volume":"5","author":"L Wang","year":"2023","journal-title":"Front Comput Sci"},{"key":"ref34","doi-asserted-by":"crossref","first-page":"100041","DOI":"10.1016\/j.yjbinx.2019.100041","article-title":"Critical care information display approaches and design frameworks: a systematic review and meta-analysis","volume":"3","author":"M C Wright","year":"2019","journal-title":"J Biomed Inform X"},{"key":"ref35","doi-asserted-by":"crossref","first-page":"103839","DOI":"10.1016\/j.artint.2022.103839","article-title":"Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making","volume":"316","author":"O Wysocki","year":"2023","journal-title":"Artif Intell"},{"key":"ref36","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.inffus.2023.03.008","article-title":"A systematic review of trustworthy and explainable artificial intelligence in healthcare: assessment of quality, bias risk, and data fusion","volume":"96","author":"A S Albahri","year":"2023","journal-title":"Inf Fusion"},{"issue":"02","key":"ref37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3205942","article-title":"Introduction to the special issue on human-centered machine learning","volume":"8","author":"R Fiebrink","year":"2018","journal-title":"ACM Trans Interact Intell Syst"},{"key":"ref38","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.neucom.2017.01.105","article-title":"What you see is what you can change: human-centered machine learning by interactive visualization","volume":"268","author":"D Sacha","year":"2017","journal-title":"Neurocomputing"},{"issue":"02","key":"ref39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3185517","article-title":"A review of user interface design for interactive machine learning","volume":"8","author":"J J Dudley","year":"2018","journal-title":"ACM Trans Interact Intell Syst"},{"issue":"03","key":"ref40","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1111\/cgf.14329","article-title":"A survey of human-centered evaluations in human-centered machine learning","volume":"40","author":"F Sperrle","year":"2021","journal-title":"Comput Graph Forum"},{"issue":"01","key":"ref41","doi-asserted-by":"crossref","first-page":"488","DOI":"10.1177\/1071181319631248","article-title":"A human factors engineering education perspective on data science, machine learning and automation","volume":"63","author":"D Hannon","year":"2019","journal-title":"Proc Hum Factors Ergon Soc Annu Meet"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/a-2618-4470.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:19:04Z","timestamp":1760138344000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/a-2618-4470"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8]]},"references-count":37,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2025,8,6]]},"published-print":{"date-parts":[[2025,8]]}},"URL":"https:\/\/doi.org\/10.1055\/a-2618-4470","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8]]}}}