{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T06:04:11Z","timestamp":1789365851335,"version":"build-2803163510"},"reference-count":58,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T00:00:00Z","timestamp":1789344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>Healthcare artificial intelligence (AI) is commonly evaluated using measures of discrimination, calibration, sensitivity, specificity, and benchmark accuracy. Although these metrics are necessary, safety ultimately depends on what occurs after an AI output enters a clinical system, whether it is noticed, trusted, verified, acted upon, and propagated through existing workflows and organizational capacities. Evidence from postmarket reports, human-AI studies, drift analyses, and equity audits demonstrates that hazards can arise at multiple points along this pathway, while remaining insufficient to quantify the incidence, attributable severity, or long-term consequences of AI-related harm. We therefore propose a five-stage sociotechnical pathway that traces risk from upstream vulnerability through AI behavior, human-workflow mediation, decision or system effect, and downstream harm. Rather than adding another catalog of governance principles, the framework follows how a specific vulnerability propagates and treats each transition as an auditable control point linked to measurable indicators, accountable actors, escalation criteria, and response actions. Thresholds should be prespecified according to the intended use and local context.<\/jats:p>","DOI":"10.3389\/frai.2026.1956533","type":"journal-article","created":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T05:39:45Z","timestamp":1789364385000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["From model failure to system harm: operationalizing a sociotechnical pathway for healthcare AI safety"],"prefix":"10.3389","volume":"9","author":[{"given":"Burhan","family":"Sebin","sequence":"first","affiliation":[{"name":"eMerge Americas","place":["Miami, FL, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irem","family":"Karaman Sebin","sequence":"additional","affiliation":[{"name":"Baptist Health South Florida","place":["Miami, FL, United States"]},{"name":"Herbert Wertheim College of Medicine, Florida International University","place":["Miami, FL, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,14]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"3032","DOI":"10.1038\/s41591-026-04503-6","article-title":"Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial","volume":"32","author":"Agweyu","year":"","journal-title":"Nat. Med."},{"key":"B2","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1038\/s44360-026-00082-5","article-title":"Safety of a large language model-based clinical decision support system in African primary healthcare","volume":"1","author":"Agweyu","year":"","journal-title":"Nat. Health"},{"key":"B3","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1186\/s12911-017-0430-8","article-title":"Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system","volume":"17","author":"Ancker","year":"2017","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"B4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3313831.3376718","article-title":"\u201cA human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy,\u201d","author":"Beede","year":"2020","journal-title":"Proceedings of the 2020 CHI conference on human factors in computing systems"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1152","DOI":"10.1038\/s41591-026-04229-5","article-title":"LLM-assisted systematic review of large language models in clinical medicine","volume":"32","author":"Chen","year":"2026","journal-title":"Nat. Med."},{"key":"B6","doi-asserted-by":"publisher","first-page":"e67552","DOI":"10.2196\/67552","article-title":"Regulatory insights from 27 years of artificial intelligence\/machine learning-enabled medical device recalls in the United States: implications for future governance","volume":"13","author":"Chen","year":"2025","journal-title":"JMIR Med. Inform."},{"key":"B7","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1038\/s41591-020-1037-7","article-title":"Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension","volume":"26","author":"Cruz Rivera","year":"2020","journal-title":"Nat. Med."},{"key":"B8","doi-asserted-by":"publisher","first-page":"40082","DOI":"10.57187\/smw.2023.40082","article-title":"Tackling alert fatigue with a semi-automated clinical decision support system: quantitative evaluation and end-user survey","volume":"153","author":"Dahmke","year":"2023","journal-title":"Swiss Med. Wkly."},{"key":"B9","doi-asserted-by":"publisher","first-page":"e222176","DOI":"10.1148\/radiol.222176","article-title":"Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance","volume":"307","author":"Dratsch","year":"2023","journal-title":"Radiology"},{"key":"B10","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1038\/s41746-021-00385-9","article-title":"Do as AI say: susceptibility in deployment of clinical decision-aids","volume":"4","author":"Gaube","year":"2021","journal-title":"NPJ Digit. Med."},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-026-03073-8","article-title":"Operationalizing the EU AI Act in a comprehensive cancer center through an institutional governance framework","author":"Gehin","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B12","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat. Med."},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-026-02837-6","article-title":"Clinical outcomes and reporting quality of large language model interventions in practice: a systematic evidence map","author":"He","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B14","doi-asserted-by":"publisher","first-page":"1669","DOI":"10.1080\/00140139.2013.838643","article-title":"SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients","volume":"56","author":"Holden","year":"2013","journal-title":"Ergonomics"},{"key":"B15","doi-asserted-by":"publisher","first-page":"899","DOI":"10.14309\/ajg.0000000000003955","article-title":"Computer-aided colonoscopy alert fatigue and its effect on adenoma detection","volume":"121","author":"Huang","year":"2026","journal-title":"Am. J. Gastroenterol."},{"key":"B16","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1038\/s41746-026-02418-7","article-title":"Advancing healthcare AI governance through a comprehensive maturity model based on systematic review","volume":"9","author":"Hussein","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B17","doi-asserted-by":"publisher","first-page":"2275","DOI":"10.1001\/jama.2023.22295","article-title":"Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study","volume":"330","author":"Jabbour","year":"2023","journal-title":"JAMA"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3310\/TATM3277","article-title":"Implementing an artificial intelligence command centre in the NHS: a mixed-methods study","volume":"12","author":"Johnson","year":"2024","journal-title":"Health Soc. Care Deliv. Res."},{"key":"B19","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s11547-025-01964-6","article-title":"Automation bias in AI-assisted detection of cerebral aneurysms on time-of-flight MR angiography","volume":"130","author":"Kim","year":"2025","journal-title":"Radiol. Med."},{"key":"B20","doi-asserted-by":"publisher","first-page":"e252717","DOI":"10.1001\/jamahealthforum.2025.2717","article-title":"Performance drift in a nationally deployed population health risk algorithm in the US Veterans Health Administration","volume":"6","author":"Kolla","year":"2025","journal-title":"JAMA Health Forum"},{"key":"B21","doi-asserted-by":"publisher","first-page":"298","DOI":"10.3233\/SHTI240871","article-title":"Automation bias in AI-decision support: results from an empirical study","volume":"317","author":"K\u00fccking","year":"2024","journal-title":"Stud. Health Technol. Inform."},{"key":"B22","doi-asserted-by":"publisher","first-page":"e2549963","DOI":"10.1001\/jamanetworkopen.2025.49963","article-title":"Vulnerability of large language models to prompt injection when providing medical advice","volume":"8","author":"Lee","year":"2025","journal-title":"JAMA Netw. Open"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2024-081554","article-title":"FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare","author":"Lekadir","year":"2025","journal-title":"BMJ"},{"key":"B24","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1038\/s41746-026-02712-4","article-title":"The absence of full lifecycle risk management for AI-based medical devices in radiology","volume":"9","author":"Li","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B25","doi-asserted-by":"publisher","first-page":"e253351","DOI":"10.1001\/jamahealthforum.2025.3351","article-title":"Benefit-risk reporting for FDA-cleared artificial intelligence-enabled medical devices","volume":"6","author":"Lin","year":"2025","journal-title":"JAMA Health Forum"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1364","DOI":"10.1038\/s41591-020-1034-x","article-title":"Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension","volume":"26","author":"Liu","year":"2020","journal-title":"Nat. Med."},{"key":"B27","doi-asserted-by":"publisher","first-page":"e2417641","DOI":"10.1001\/jamanetworkopen.2024.17641","article-title":"Performance of large language models on medical oncology examination questions","volume":"7","author":"Longwell","year":"2024","journal-title":"JAMA Netw. Open"},{"key":"B28","doi-asserted-by":"publisher","first-page":"943768","DOI":"10.3389\/fdgth.2022.943768","article-title":"Considerations in the reliability and fairness audits of predictive models for advance care planning","volume":"4","author":"Lu","year":"2022","journal-title":"Front. Digit. Health"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1227","DOI":"10.1093\/jamia\/ocad065","article-title":"More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA","volume":"30","author":"Lyell","year":"2023","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B30","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1136\/bmjqs-2019-009484","article-title":"Governing the safety of artificial intelligence in healthcare","volume":"28","author":"Macrae","year":"2019","journal-title":"BMJ Qual. Saf."},{"key":"B31","doi-asserted-by":"publisher","first-page":"1999","DOI":"10.1111\/risa.13850","article-title":"Learning from the failure of autonomous and intelligent systems: accidents, safety, and sociotechnical sources of risk","volume":"42","author":"Macrae","year":"2022","journal-title":"Risk Anal."},{"key":"B32","doi-asserted-by":"publisher","first-page":"1324","DOI":"10.1093\/jac\/dkaf077","article-title":"GP or ChatGPT? Ability of large language models (LLMs) to support general practitioners when prescribing antibiotics","volume":"80","author":"Ngoc Nguyen","year":"2025","journal-title":"J. Antimicrob. Chemother"},{"key":"B33","doi-asserted-by":"publisher","first-page":"e70176","DOI":"10.2196\/70176","article-title":"Reducing hallucinations and trade-offs in responses in generative AI chatbots for cancer information: development and evaluation study","volume":"11","author":"Nishisako","year":"2025","journal-title":"JMIR Cancer"},{"key":"B34","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"key":"B35","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1038\/s43856-025-01021-3","article-title":"Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support","volume":"5","author":"Omar","year":"2025","journal-title":"Commun. Med. (Lond.)"},{"key":"B36","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1038\/s41746-026-02705-3","article-title":"Framework for artificial intelligence implementation research in healthcare: synthesizing current evidence on barriers and facilitators","volume":"9","author":"Powis","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B37","doi-asserted-by":"publisher","first-page":"e2617920","DOI":"10.1001\/jamanetworkopen.2026.17920","article-title":"Clinical evidence and FDA recalls of artificial intelligence-enabled medical devices","volume":"9","author":"Ren","year":"2026","journal-title":"JAMA Netw. Open"},{"key":"B38","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1177\/15553434221097357","article-title":"A sociotechnical systems framework for the application of artificial intelligence in health care delivery","volume":"16","author":"Salwei","year":"2022","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"B39","doi-asserted-by":"publisher","first-page":"2098","DOI":"10.1038\/s41379-021-00859-x","article-title":"Quality control stress test for deep learning-based diagnostic model in digital pathology","volume":"34","author":"Sch\u00f6mig-Markiefka","year":"2021","journal-title":"Mod. Pathol."},{"key":"B40","doi-asserted-by":"publisher","first-page":"2176","DOI":"10.1038\/s41591-021-01595-0","article-title":"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations","volume":"27","author":"Seyyed-Kalantari","year":"2021","journal-title":"Nat. Med."},{"key":"B41","doi-asserted-by":"publisher","first-page":"i68","DOI":"10.1136\/qshc.2010.042085","article-title":"A new sociotechnical model for studying health information technology in complex adaptive healthcare systems","volume":"19","author":"Sittig","year":"2010","journal-title":"Qual. Saf. Health Care"},{"key":"B42","doi-asserted-by":"publisher","first-page":"2477","DOI":"10.1056\/NEJMc2029240","article-title":"Racial bias in pulse oximetry measurement","volume":"383","author":"Sjoding","year":"2020","journal-title":"N. Engl. J. Med."},{"key":"B43","article-title":"\u201cArtificial intelligence-enabled device software functions: lifecycle management and marketing submission recommendations,\u201d","volume-title":"Draft Guidance for Industry and Food and Drug Administration Staff","year":"2025"},{"key":"B44","doi-asserted-by":"publisher","first-page":"569","DOI":"10.1038\/s41746-026-02971-1","article-title":"Meaningful oversight of medical AI beyond human in the loop","volume":"9","author":"van de Sande","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B45","doi-asserted-by":"publisher","first-page":"1503","DOI":"10.1093\/jamia\/ocad088","article-title":"Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework","volume":"30","author":"van der Vegt","year":"2023","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B46","doi-asserted-by":"publisher","first-page":"e070904","DOI":"10.1136\/bmj-2022-070904","article-title":"Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"377","author":"Vasey","year":"2022","journal-title":"BMJ"},{"key":"B47","doi-asserted-by":"publisher","first-page":"1752","DOI":"10.1016\/j.jaci.2025.08.012","article-title":"Impact of race-based spirometry equations on guideline-based asthma management in clinical trials","volume":"156","author":"Visness","year":"2025","journal-title":"J. Allergy Clin. Immunol."},{"key":"B48","doi-asserted-by":"publisher","first-page":"516","DOI":"10.1038\/s41746-026-02679-2","article-title":"Governance for safe and responsible AI in healthcare organisations: a scoping review of frameworks","volume":"9","author":"Wang","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B49","doi-asserted-by":"publisher","first-page":"1684","DOI":"10.1093\/jamia\/ocad118","article-title":"Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias","volume":"30","author":"Wang","year":"2023","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"B50","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/s41746-026-02382-2","article-title":"Human-large language model collaboration in clinical medicine: a systematic review and meta-analysis","volume":"9","author":"Wang","year":"2026","journal-title":"NPJ Digit. Med."},{"key":"B51","doi-asserted-by":"publisher","first-page":"e47125","DOI":"10.2196\/47125","article-title":"Evaluating algorithmic bias in 30-day hospital readmission models: retrospective analysis","volume":"26","author":"Wang","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"B52","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1038\/s41746-025-01900-y","article-title":"A practical framework for appropriate implementation and review of artificial intelligence (FAIR-AI) in healthcare","volume":"8","author":"Wells","year":"2025","journal-title":"NPJ Digit. Med."},{"key":"B53","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern. Med."},{"key":"B54","year":"2021","journal-title":"Ethics and Governance of Artificial Intelligence for Health: WHO Guidance"},{"key":"B55","first-page":"1414","article-title":"Analyzing and mitigating model drift in acute kidney injury prediction for hospitalized patients","volume":"2024","author":"Xu","year":"2025","journal-title":"AMIA Annu. Symp. Proc."},{"key":"B56","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1038\/s41746-025-01506-4","article-title":"Clinical trials informed framework for real world clinical implementation and deployment of artificial intelligence applications","volume":"8","author":"You","year":"2025","journal-title":"NPJ Digit. Med."},{"key":"B57","doi-asserted-by":"publisher","first-page":"103977","DOI":"10.1016\/j.jbi.2021.103977","article-title":"Membership inference attacks against synthetic health data","volume":"125","author":"Zhang","year":"2022","journal-title":"J. Biomed. Inform."},{"key":"B58","doi-asserted-by":"publisher","first-page":"7281","DOI":"10.1038\/s41467-021-27577-x","article-title":"A machine and human reader study on AI diagnosis model safety under attacks of adversarial images","volume":"12","author":"Zhou","year":"2021","journal-title":"Nat. Commun."}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1956533\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T05:39:46Z","timestamp":1789364386000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1956533\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,14]]},"references-count":58,"alternative-id":["10.3389\/frai.2026.1956533"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1956533","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,14]]},"article-number":"1956533"}}