{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T20:36:56Z","timestamp":1793479016414,"version":"4.5.0"},"reference-count":64,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T00:00:00Z","timestamp":1740355200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>The safety of Artificial Intelligence (AI) systems is as much one of human decision-making as a technological question. In AI-driven decision support systems, particularly in high-stakes settings such as healthcare, ensuring the safety of human-AI interactions is paramount, given the potential risks of following erroneous AI recommendations. To explore this question, we ran a safety-focused clinician-AI interaction study in a physical simulation suite. Physicians were placed in a simulated intensive care ward, with a human nurse (played by an experimenter), an ICU data chart, a high-fidelity patient mannequin and an AI recommender system on a display. Clinicians were asked to prescribe two drugs for the simulated patients suffering from sepsis and wore eye-tracking glasses to allow us to assess where their gaze was directed. We recorded clinician treatment plans before and after they saw the AI treatment recommendations, which could be either \u2018safe\u2019 or \u2018unsafe\u2019. 92% of clinicians rejected unsafe AI recommendations vs 29% of safe ones. Physicians paid increased attention (+37% gaze fixations) to unsafe AI recommendations vs safe ones. However, visual attention on AI explanations was not greater in unsafe scenarios. Similarly, clinical information (patient monitor, patient chart) did not receive more attention after an unsafe versus safe AI reveal suggesting that the physicians did not look back to these sources of information to investigate why the AI suggestion might be unsafe. Physicians were only successfully persuaded to change their dose by scripted comments from the bedside nurse 5% of the time. Our study emphasises the importance of human oversight in safety-critical AI and the value of evaluating human-AI systems in high-fidelity settings that more closely resemble real world practice.<\/jats:p>","DOI":"10.1371\/journal.pdig.0000726","type":"journal-article","created":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T13:31:25Z","timestamp":1740403885000},"page":"e0000726","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":25,"title":["Safety of human-AI cooperative decision-making within intensive care: A physical simulation study"],"prefix":"10.1371","volume":"4","author":[{"given":"Paul","family":"Festor","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Myura","family":"Nagendran","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anthony C.","family":"Gordon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0813-7207","authenticated-orcid":true,"given":"Aldo A.","family":"Faisal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthieu","family":"Komorowski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,2,24]]},"reference":[{"key":"pdig.0000726.ref001","doi-asserted-by":"crossref","first-page":"2633","DOI":"10.1038\/s41591-023-02552-9","article-title":"Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial","volume":"29","author":"G Wang","year":"2023","journal-title":"Nat Med"},{"issue":"4","key":"pdig.0000726.ref002","first-page":"4680","article-title":"DeepThermal: combustion optimization for thermal power generating units using offline reinforcement learning","volume":"36","author":"X Zhan","year":"2022","journal-title":"Proc AAAI Conf Artif Intell"},{"issue":"11","key":"pdig.0000726.ref003","doi-asserted-by":"crossref","first-page":"1463","DOI":"10.1038\/s41562-021-01226-1","article-title":"Better science for better decision-making in future crises","volume":"5","author":"SA Galea","year":"2021","journal-title":"Nat Hum Behav"},{"issue":"1","key":"pdig.0000726.ref004","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s41746-023-00955-z","article-title":"Quantifying the impact of AI recommendations with explanations on prescription decision making","volume":"6","author":"M Nagendran","year":"2023","journal-title":"NPJ Digital Med"},{"issue":"6","key":"pdig.0000726.ref005","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1038\/s41562-021-01128-2","article-title":"Bad machines corrupt good morals","volume":"5","author":"N K\u00f6bis","year":"2021","journal-title":"Nat Hum Behav"},{"key":"pdig.0000726.ref006","first-page":"291","volume-title":"Levels of Autonomy and Safety Assurance for AI-Based Clinical Decision Systems","author":"P Festor","year":"2021"},{"key":"pdig.0000726.ref007","unstructured":"European Parliament and Council. Regulation (EU) 2024\/Artificial Intelligence Act [Internet]. 2024. Available from: https:\/\/artificialintelligenceact.eu\/wp-content\/uploads\/2024\/01\/AI-Act-FullText.pdf"},{"key":"pdig.0000726.ref008","article-title":"Learning to influence human behavior with offline reinforcement learning","volume":"36","author":"J Hong","year":"2023","journal-title":"Advances in neural information processing systems"},{"key":"pdig.0000726.ref009","article-title":"Deep reinforcement learning from human preferences","volume":"30","author":"PF Christiano","year":"2017","journal-title":"Advances in neural information processing systems"},{"key":"pdig.0000726.ref010","volume-title":"Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI","author":"RM Monarch","year":"2021"},{"issue":"12","key":"pdig.0000726.ref011","doi-asserted-by":"crossref","first-page":"1636","DOI":"10.1038\/s41562-021-01146-0","article-title":"Understanding, explaining, and utilizing medical artificial intelligence","volume":"5","author":"R Cadario","year":"2021","journal-title":"Nat Hum Behav"},{"key":"pdig.0000726.ref012","first-page":"1","volume-title":"2020 IEEE International conference on healthcare informatics (ICHI)","author":"Y Jia","year":"2020"},{"issue":"1","key":"pdig.0000726.ref013","doi-asserted-by":"crossref","first-page":"e100549","DOI":"10.1136\/bmjhci-2022-100549","article-title":"Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment","volume":"29","author":"P Festor","year":"2022","journal-title":"BMJ Health Care Inform"},{"key":"pdig.0000726.ref014","first-page":"887","volume-title":"Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning","author":"X Peng","year":"2018"},{"issue":"1","key":"pdig.0000726.ref015","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.annemergmed.2021.02.006","article-title":"Early care of adults with suspected sepsis in the emergency department and out-of-hospital environment: a consensus-based task force report","volume":"78","author":"DM Yealy","year":"2021","journal-title":"Ann Emerg Med"},{"key":"pdig.0000726.ref016","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1016\/j.jcrc.2021.10.008","article-title":"Monitoring, management, and outcome of hypotension in intensive care unit patients, an international survey of the European Society of Intensive Care Medicine","volume":"67","author":"W Van der Ven","year":"2022","journal-title":"J Crit Care"},{"issue":"2","key":"pdig.0000726.ref017","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1097\/CCM.0000000000006135","article-title":"Surviving sepsis campaign research priorities 2023","volume":"52","author":"Surviving Sepsis Campaign Research Committee","year":"2024","journal-title":"Crit Care Med"},{"issue":"10219","key":"pdig.0000726.ref018","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/S0140-6736(19)32989-7","article-title":"Global, regional, and national sepsis incidence and mortality, 1990-2017: analysis for the Global Burden of Disease Study","volume":"395","author":"KE Rudd","year":"2020","journal-title":"Lancet (London, England)"},{"issue":"6","key":"pdig.0000726.ref019","doi-asserted-by":"crossref","first-page":"e15154","DOI":"10.2196\/15154","article-title":"Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians","volume":"22","author":"O Asan","year":"2020","journal-title":"J Med Internet Res"},{"issue":"7","key":"pdig.0000726.ref020","doi-asserted-by":"crossref","first-page":"750","DOI":"10.1007\/s00134-021-06446-7","article-title":"Moving from bytes to bedside: a systematic review on the use of artificial intelligence in the intensive care unit","volume":"47","author":"D Van de Sande","year":"2021","journal-title":"Intensive Care Med"},{"key":"pdig.0000726.ref021","doi-asserted-by":"crossref","first-page":"102861","DOI":"10.1016\/j.artmed.2024.102861","article-title":"Challenges and strategies for wide-scale artificial intelligence (AI) deployment in healthcare practices: a perspective for healthcare organizations","volume":"151","author":"P Esmaeilzadeh","year":"2024","journal-title":"Artif Intell Med"},{"key":"pdig.0000726.ref022","doi-asserted-by":"crossref","first-page":"m4786","DOI":"10.1136\/bmj.m4786","article-title":"Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial","volume":"372","author":"FP Wilson","year":"2021","journal-title":"BMJ"},{"issue":"1","key":"pdig.0000726.ref023","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1186\/s12911-016-0377-1","article-title":"Cognitive biases associated with medical decisions: a systematic review","volume":"16","author":"G Saposnik","year":"2016","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"pdig.0000726.ref024","doi-asserted-by":"crossref","first-page":"e100081","DOI":"10.1136\/bmjhci-2019-100081","article-title":"Human factors challenges for the safe use of artificial intelligence in patient care","volume":"26","author":"M Sujan","year":"2019","journal-title":"BMJ Health Care informat"},{"issue":"3","key":"pdig.0000726.ref025","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/BF02596149","article-title":"Systematic errors in medical decision making: judgment limitations","volume":"2","author":"NV Dawson","year":"1987","journal-title":"J Gen Intern Med"},{"issue":"12","key":"pdig.0000726.ref026","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1001\/archinte.163.12.1409","article-title":"Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review","volume":"163","author":"R Kaushal","year":"2003","journal-title":"Arch Intern Med"},{"key":"pdig.0000726.ref027","volume-title":"Towards ecological validity in evaluating uncertainty","author":"PS Quinan","year":"2015"},{"key":"pdig.0000726.ref028","article-title":"Predict responsibly: improving fairness and accuracy by learning to defer","volume":"31","author":"D Madras","year":"2018","journal-title":"Advances in neural information processing systems"},{"issue":"1","key":"pdig.0000726.ref029","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1097\/CNQ.0b013e3181c8dfd4","article-title":"Use of Simulation Training in the Intensive Care Unit","volume":"33","author":"DL Cato","year":"2010","journal-title":"Crit Care Nurs Q"},{"issue":"2","key":"pdig.0000726.ref030","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1111\/ejn.15595","article-title":"Ecological validity in exercise neuroscience research: A systematic investigation","volume":"55","author":"M Chang","year":"2022","journal-title":"Eur J Neurosci"},{"key":"pdig.0000726.ref031","unstructured":"EU AI Act Article 8 [Internet]. Artificial Intelligence Act. [cited 2023 Sep 29]. Available from: https:\/\/artificialintelligenceact.com\/title-iii\/chapter-2\/article-8\/"},{"key":"pdig.0000726.ref032","unstructured":"GOV.UK [Internet]. Software and AI as a Medical Device Change Programme. 2023 [cited 2023 Oct 2]. Available from: https:\/\/www.gov.uk\/government\/publications\/software-and-ai-as-a-medical-device-change-programme"},{"key":"pdig.0000726.ref033","unstructured":"Fiannaca A, Bennett C, Kane S, Morris MR. Beyond safety: toward a value-sensitive approach to the design of AI systems. 2022"},{"issue":"1","key":"pdig.0000726.ref034","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1038\/s41746-024-01200-x","article-title":"Eye tracking insights into physician behaviour with safe and unsafe explainable AI recommendations","volume":"7","author":"M Nagendran","year":"2024","journal-title":"NPJ Digit Med"},{"issue":"4","key":"pdig.0000726.ref035","doi-asserted-by":"crossref","first-page":"396","DOI":"10.9734\/BJAST\/2015\/14975","article-title":"Likert scale: Explored and explained","volume":"7","author":"A Joshi","year":"2015","journal-title":"Br J Appl Sci Technol"},{"key":"pdig.0000726.ref036","article-title":"A survey of explainable reinforcement learning","author":"S Milani","year":"2022","journal-title":"arXiv preprint arXiv:2202.08434"},{"key":"pdig.0000726.ref037","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1007\/978-1-0716-2391-6_4","volume-title":"Eye Tracking: Background, Methods, and Applications","author":"JA Harston","year":"2022"},{"issue":"1","key":"pdig.0000726.ref038","doi-asserted-by":"crossref","DOI":"10.16910\/jemr.6.1.3","article-title":"Using eye tracking to trace a cognitive process: Gaze behaviour during decision making in a natural environment","volume":"6","author":"K Gidl\u00f6f","year":"2013","journal-title":"J Eye Mov Res"},{"key":"pdig.0000726.ref039","unstructured":"Women in Intensive Care Medicine | The Faculty of Intensive Care Medicine [Internet]. [cited 2023 Jan 4]. Available from: https:\/\/www.ficm.ac.uk\/careersworkforceworkforce\/women-in-intensive-care-medicine"},{"issue":"1","key":"pdig.0000726.ref040","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.obhdp.2003.08.002","article-title":"Receiving other people\u2019s advice: Influence and benefit","volume":"93","author":"I Yaniv","year":"2004","journal-title":"Organ Behav Hum Decis Process"},{"key":"pdig.0000726.ref041","article-title":"The response shift paradigm to quantify human trust in AI recommendations.","author":"A Shafti","year":"2022"},{"key":"pdig.0000726.ref042","article-title":"From human-computer interaction to human-AI Interaction: new challenges and opportunities for enabling human-centered AI","volume":"5","author":"W Xu","year":"2021"},{"key":"pdig.0000726.ref043","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency [Internet]","author":"Y Zhang","year":"2020"},{"issue":"1","key":"pdig.0000726.ref044","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s12910-024-01062-8","article-title":"Ethical considerations and concerns in the implementation of AI in pharmacy practice: a cross-sectional study","volume":"25","author":"HE Hasan","year":"2024","journal-title":"BMC Med Ethics"},{"issue":"CSCW2","key":"pdig.0000726.ref045","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3555572","article-title":"Understanding User Reliance on AI in Assisted Decision-Making","volume":"6","author":"S Cao","year":"2022","journal-title":"Proc ACM Hum-Comput Interact"},{"issue":"1","key":"pdig.0000726.ref046","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-021-00385-9","article-title":"Do as AI say: susceptibility in deployment of clinical decision-aids","volume":"4","author":"S Gaube","year":"2021","journal-title":"NPJ Digit Med"},{"issue":"23","key":"pdig.0000726.ref047","doi-asserted-by":"crossref","first-page":"2275","DOI":"10.1001\/jama.2023.22295","article-title":"Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study","volume":"330","author":"S Jabbour","year":"2023","journal-title":"JAMA"},{"issue":"9","key":"pdig.0000726.ref048","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1136\/bmjqs-2013-002165","article-title":"Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review","volume":"23","author":"SR Ranji","year":"Sep","journal-title":"BMJ Qual Saf"},{"key":"pdig.0000726.ref049","doi-asserted-by":"crossref","unstructured":"Catling FJR, Nagendran M, Festor P, Bien Z, Harris S, Faisal AA, et al. Data-driven decision support for individualised cardiovascular resuscitation in sepsis: a scoping review and primer for clinicians. 2023;2023-08. medRxiv.","DOI":"10.1101\/2023.08.26.23294666"},{"issue":"7","key":"pdig.0000726.ref050","doi-asserted-by":"crossref","first-page":"e18477","DOI":"10.2196\/18477","article-title":"Reinforcement learning for clinical decision support in critical care: comprehensive review","volume":"22","author":"S Liu","year":"2020","journal-title":"J Med Internet Res"},{"issue":"11","key":"pdig.0000726.ref051","doi-asserted-by":"crossref","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","article-title":"The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care","volume":"24","author":"M Komorowski","year":"2018","journal-title":"Nat Med"},{"issue":"1","key":"pdig.0000726.ref052","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-022-00708-4","article-title":"An interpretable RL framework for pre-deployment modeling in ICU hypotension management","volume":"5","author":"K Zhang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"pdig.0000726.ref053","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780199936472.001.0001","volume-title":"Epistemic authority: A theory of trust, authority, and autonomy in belief","author":"LT Zagzebski","year":"2012"},{"key":"pdig.0000726.ref054","volume-title":"Testimony: A philosophical study","author":"CAJ Coady","year":"1992"},{"issue":"9","key":"pdig.0000726.ref055","doi-asserted-by":"crossref","first-page":"e0203629","DOI":"10.1371\/journal.pone.0203629","article-title":"Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze","volume":"13","author":"K Krejtz","year":"2018","journal-title":"PLoS One"},{"key":"pdig.0000726.ref056","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2993901.2993908","volume-title":"In: Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization [Internet]","author":"J Zagermann","year":"2016"},{"issue":"9","key":"pdig.0000726.ref057","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1177\/154193120605000909","article-title":"Nasa-Task Load Index (NASA-TLX); 20 Years Later","volume":"50","author":"SG Hart","year":"2006","journal-title":"Proc Hum Factors Ergon Soc Ann Meet"},{"key":"pdig.0000726.ref058","unstructured":"GOV.UK [Internet]. Crafting an intended purpose in the context of software as a medical device (SaMD). [cited 2023 Mar 28]. Available from: https:\/\/www.gov.uk\/government\/publications\/crafting-an-intended-purpose-in-the-context-of-software-as-a-medical-device-samd\/crafting-an-intended-purpose-in-the-context-of-software-as-a-medical-device-samd"},{"issue":"1","key":"pdig.0000726.ref059","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1186\/s13054-023-04393-x","article-title":"Large language models and the perils of their hallucinations","volume":"27","author":"R Azamfirei","year":"Mar","journal-title":"Crit Care"},{"issue":"CSCW1","key":"pdig.0000726.ref060","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3579460","article-title":"Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust","volume":"7","author":"N Banovic","year":"2023","journal-title":"Proc ACM Hum-Comput Interact"},{"key":"pdig.0000726.ref061","first-page":"246","volume-title":"Dataset Shift in Machine Learning","author":"J Quinonero-Candela","year":"2022"},{"key":"pdig.0000726.ref062","volume-title":"Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition","author":"P Festor","year":"2021"},{"key":"pdig.0000726.ref063","volume-title":"Evaluating Uncertainty-Based Deep Learning Explanations for Prostate Lesion Detection","author":"CM Trombley","year":"2022"},{"key":"pdig.0000726.ref064","unstructured":"Nagendran M, Festor PG, Komorowski M, Gordon AC, Faisal AA. Clinical Evaluation Framework Using Behavioural & Visual Attention Read-Outs for Explainable AI (XAI). Available from: SSRN 4496127"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0000726","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T13:31:47Z","timestamp":1740403907000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0000726"}},"subtitle":[],"editor":[{"given":"Martin G","family":"Frasch,","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,2,24]]},"references-count":64,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,2,24]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0000726","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,24]]}}}