{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T13:09:49Z","timestamp":1767704989436,"version":"3.37.3"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T00:00:00Z","timestamp":1714003200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["1R01GM137083-01"],"award-info":[{"award-number":["1R01GM137083-01"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,5,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Obtain clinicians\u2019 perspectives on early warning scores (EWS) use within context of clinical cases.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Material and Methods<\/jats:title>\n                  <jats:p>We developed cases mimicking sepsis situations. De-identified data, synthesized physician notes, and EWS representing deterioration risk were displayed in a simulated EHR for analysis. Twelve clinicians participated in semi-structured interviews to ascertain perspectives across four domains: (1) Familiarity with and understanding of artificial intelligence (AI), prediction models and risk scores; (2) Clinical reasoning processes; (3) Impression and response to EWS; and (4) Interface design. Transcripts were coded and analyzed using content and thematic analysis.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Analysis revealed clinicians have experience but limited AI and prediction\/risk modeling understanding. Case assessments were primarily based on clinical data. EWS went unmentioned during initial case analysis; although when prompted to comment on it, they discussed it in subsequent cases. Clinicians were unsure how to interpret or apply the EWS, and desired evidence on its derivation and validation. Design recommendations centered around EWS display in multi-patient lists for triage, and EWS trends within the patient record. Themes included a \u201cTrust but Verify\u201d approach to AI and early warning information, dichotomy that EWS is helpful for triage yet has disproportional signal-to-high noise ratio, and action driven by clinical judgment, not the EWS.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Clinicians were unsure of how to apply EWS, acted on clinical data, desired score composition and validation information, and felt EWS was most useful when embedded in multi-patient views. Systems providing interactive visualization may facilitate EWS transparency and increase confidence in AI-generated information.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae089","type":"journal-article","created":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T13:52:47Z","timestamp":1714053167000},"page":"1331-1340","source":"Crossref","is-referenced-by-count":6,"title":["Clinician perspectives on how situational context and augmented intelligence design features impact perceived usefulness of sepsis prediction scores embedded within a simulated electronic health record"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-9342-0523","authenticated-orcid":false,"given":"Velma L","family":"Payne","sequence":"first","affiliation":[{"name":"Kasiska Division of Health Sciences, College of Health, Idaho State University , Pocatello, ID 83209, United States"}]},{"given":"Usman","family":"Sattar","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, School of Medicine, University of Utah , Salt Lake City, UT 84108, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7121-504X","authenticated-orcid":false,"given":"Melanie","family":"Wright","sequence":"additional","affiliation":[{"name":"Tunnell Government Services, Inc. , Bethesda, MD 20817, United States"}]},{"given":"Elijah","family":"Hill","sequence":"additional","affiliation":[{"name":"Kasiska Division of Health Sciences, College of Pharmacy, Idaho State University , Pocatello, ID 83209, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4519-7997","authenticated-orcid":false,"given":"Jorie M","family":"Butler","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, School of Medicine, University of Utah , Salt Lake City, UT 84108, United States"}]},{"given":"Brekk","family":"Macpherson","sequence":"additional","affiliation":[{"name":"Virginia Commonwealth University Health System , Richmond, VA 83298, United States"}]},{"given":"Amanda","family":"Jeppesen","sequence":"additional","affiliation":[{"name":"Kasiska Division of Health Sciences, College of Pharmacy, Idaho State University , Pocatello, ID 83209, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9954-6799","authenticated-orcid":false,"given":"Guilherme","family":"Del Fiol","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, School of Medicine, University of Utah , Salt Lake City, UT 84108, United States"}]},{"given":"Karl","family":"Madaras-Kelly","sequence":"additional","affiliation":[{"name":"Kasiska Division of Health Sciences, College of Pharmacy, Idaho State University , Pocatello, ID 83209, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,4,25]]},"reference":[{"issue":"1","key":"2024052019470619100_ocae089-B1","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.resuscitation.2015.08.016","article-title":"The prevalence and significance of abnormal vital signs prior to in-hospital cardiac arrest","volume":"98","author":"Andersen","year":"2016","journal-title":"Resuscitation"},{"issue":"1","key":"2024052019470619100_ocae089-B2","doi-asserted-by":"crossref","first-page":"m1501","DOI":"10.1136\/bmj.m1501","article-title":"Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology","volume":"369","author":"Gerry","year":"2020","journal-title":"BMJ"},{"author":"National Institute for Health and Care Excellence","key":"2024052019470619100_ocae089-B3"},{"issue":"3","key":"2024052019470619100_ocae089-B4","first-page":"CD005529","article-title":"Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards","author":"McGaughey","year":"2007","journal-title":"Cochrane Database Syst Rev"},{"issue":"4","key":"2024052019470619100_ocae089-B5","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1007\/s00134-007-0532-3","article-title":"Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward","volume":"33","author":"Gao","year":"2007","journal-title":"Intensive Care Med"},{"issue":"1","key":"2024052019470619100_ocae089-B6","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1186\/s12911-020-01144-8","article-title":"Early warning score validation methodologies and performance metrics: a systematic review","volume":"20","author":"Fang","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"2024052019470619100_ocae089-B7","doi-asserted-by":"crossref","first-page":"103410","DOI":"10.1016\/j.jbi.2020.103410","article-title":"Development and validation of early warning score system: a systematic literature review","volume":"105","author":"Fu","year":"2020","journal-title":"J Biomed Inform"},{"issue":"5","key":"2024052019470619100_ocae089-B8","doi-asserted-by":"crossref","first-page":"e0217864","DOI":"10.1371\/journal.pone.0217864","article-title":"Early warning systems in obstetrics: a systematic literature review","volume":"14","author":"Umar","year":"2019","journal-title":"PLoS One"},{"key":"2024052019470619100_ocae089-B9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cmpb.2018.12.027","article-title":"Prediction of sepsis patients using machine learning approach: a meta-analysis","volume":"170","author":"Islam","year":"2019","journal-title":"Comput Methods Programs Biomed"},{"issue":"1","key":"2024052019470619100_ocae089-B10","doi-asserted-by":"crossref","first-page":"e000234","DOI":"10.1136\/bmjresp-2017-000234","article-title":"Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial","volume":"4","author":"Shimabukuro","year":"2017","journal-title":"BMJ Open Respir Res"},{"issue":"1","key":"2024052019470619100_ocae089-B11","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.compbiomed.2019.04.027","article-title":"Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs","volume":"109","author":"Barton","year":"2019","journal-title":"Comput Biol Med"},{"issue":"1","key":"2024052019470619100_ocae089-B12","doi-asserted-by":"crossref","first-page":"101820","DOI":"10.1016\/j.artmed.2020.101820","article-title":"Early detection of sepsis utilizing deep learning on electronic health record event sequences","volume":"104","author":"Lauritsen","year":"2020","journal-title":"Artif Intell Med"},{"issue":"14","key":"2024052019470619100_ocae089-B13","doi-asserted-by":"crossref","first-page":"1347","DOI":"10.1056\/NEJMra1814259","article-title":"Machine Learning in Medicine","volume":"380","author":"Rajkomar","year":"2019","journal-title":"N Engl J Med"},{"issue":"6","key":"2024052019470619100_ocae089-B14","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1001\/jama.2017.7797","article-title":"Unintended Consequences of Machine Learning in Medicine","volume":"318","author":"Cabitza","year":"2017","journal-title":"JAMA"},{"issue":"10","key":"2024052019470619100_ocae089-B15","doi-asserted-by":"crossref","first-page":"1419","DOI":"10.1093\/jamia\/ocy068","article-title":"Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review","volume":"25","author":"Xiao","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024052019470619100_ocae089-B16","doi-asserted-by":"crossref","first-page":"100485","DOI":"10.1016\/j.xcrm.2021.100485","article-title":"The inclusion of augmented intelligence in medicine: a framework for successful implementation","volume":"3","author":"Bazoukis","year":"2022","journal-title":"Cell Rep Med"},{"issue":"3","key":"2024052019470619100_ocae089-B17","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1136\/medethics-2019-105586","article-title":"On the ethics of algorithmic decision-making in healthcare","volume":"46","author":"Grote","year":"2020","journal-title":"J Med Ethics"},{"issue":"18","key":"2024052019470619100_ocae089-B18","doi-asserted-by":"crossref","first-page":"1765","DOI":"10.1001\/jama.2019.15064","article-title":"Potential liability for physicians using artificial intelligence","volume":"322","author":"Price","year":"2019","journal-title":"JAMA"},{"issue":"5","key":"2024052019470619100_ocae089-B19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/gigascience\/gix019","article-title":"The need to approximate the use-case in clinical machine learning","volume":"6","author":"Saeb","year":"2017","journal-title":"Gigascience"},{"issue":"3","key":"2024052019470619100_ocae089-B20","doi-asserted-by":"crossref","first-page":"ooab040","DOI":"10.1093\/jamiaopen\/ooab040","article-title":"A simple electronic medical record system designed for research","volume":"4","author":"King","year":"2021","journal-title":"JAMIA Open"},{"issue":"9","key":"2024052019470619100_ocae089-B21","doi-asserted-by":"crossref","first-page":"1339","DOI":"10.1097\/CCM.0000000000005492","article-title":"The impact of a machine learning early warning score on hospital mortality: a multicenter clinical intervention trial","volume":"50","author":"Winslow","year":"2022","journal-title":"Crit Care Med"},{"journal-title":"medRxiv","article-title":"Multicenter development and prospective validation of eCARTv5: a gradient boosted machine learning early warning score","author":"Churpek","key":"2024052019470619100_ocae089-B22"},{"issue":"6","key":"2024052019470619100_ocae089-B23","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1164\/rccm.201406-1022OC","article-title":"Multicenter development and validation of a risk stratification tool for ward patients","volume":"190","author":"Churpek","year":"2014","journal-title":"Am J Respir Crit Care Med"},{"issue":"11","key":"2024052019470619100_ocae089-B24","doi-asserted-by":"crossref","first-page":"1805","DOI":"10.1097\/CCM.0000000000002648","article-title":"Investigating the impact of different suspicion of infection criteria on the accuracy of quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores","volume":"45","author":"Churpek","year":"2017","journal-title":"Crit Care Med"},{"issue":"1","key":"2024052019470619100_ocae089-B25","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1186\/s43058-021-00202-0","article-title":"Comparing inductive and deductive analysis techniques to understand health service implementation problems: a case study of childhood vaccination barriers","volume":"2","author":"Bonner","year":"2021","journal-title":"Implement Sci Commun"},{"issue":"1","key":"2024052019470619100_ocae089-B26","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1177\/160940690600500107","article-title":"Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development","volume":"5","author":"Fereday","year":"2006","journal-title":"Int J Qual Methods"},{"year":"2023","author":"Lumivero","key":"2024052019470619100_ocae089-B27"},{"issue":"6","key":"2024052019470619100_ocae089-B28","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1055\/a-2142-9325","article-title":"Challenges and ethical considerations to successfully implement artificial intelligence in clinical medicine and neuroscience: a narrative review","volume":"56","author":"Monteith","year":"2023","journal-title":"Pharmacopsychiatry"},{"issue":"2","key":"2024052019470619100_ocae089-B29","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/s11739-020-02580-0","article-title":"Diagnostic reasoning in internal medicine: a practical reappraisal","volume":"16","author":"Corazza","year":"2021","journal-title":"Intern Emerg Med"},{"issue":"1","key":"2024052019470619100_ocae089-B30","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1186\/s41235-022-00460-z","article-title":"The challenge of cognitive science for medical diagnosis","volume":"8","author":"Croskerry","year":"2023","journal-title":"Cogn Res Princ Implic"},{"issue":"3","key":"2024052019470619100_ocae089-B31","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1093\/jamia\/ocaa296","article-title":"Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: a scoping review","volume":"28","author":"Schwartz","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024052019470619100_ocae089-B32","doi-asserted-by":"crossref","first-page":"104643","DOI":"10.1016\/j.ijmedinf.2021.104643","article-title":"Engaging clinicians early during the development of a graphical user display of an intelligent alerting system at the bedside","volume":"159","author":"Helman","year":"2022","journal-title":"Int J Med Inform"},{"issue":"4","key":"2024052019470619100_ocae089-B33","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1055\/s-0043-1775565","article-title":"Engaging multidisciplinary clinical users in the design of an artificial intelligence-powered graphical user interface for intensive care unit instability decision support","volume":"14","author":"Helman","year":"2023","journal-title":"Appl Clin Inform"},{"issue":"23","key":"2024052019470619100_ocae089-B34","doi-asserted-by":"crossref","first-page":"2275","DOI":"10.1001\/jama.2023.22295","article-title":"Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study","volume":"330","author":"Jabbour","year":"2023","journal-title":"JAMA"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1331\/57769072\/ocae089.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1331\/57769072\/ocae089.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,20]],"date-time":"2024-05-20T19:48:14Z","timestamp":1716234494000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/6\/1331\/7658195"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,25]]},"references-count":34,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,4,25]]},"published-print":{"date-parts":[[2024,5,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae089","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"type":"print","value":"1067-5027"},{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2024,6,1]]},"published":{"date-parts":[[2024,4,25]]}}}