{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T18:07:42Z","timestamp":1789668462112,"version":"4.0.1"},"reference-count":36,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T00:00:00Z","timestamp":1789603200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01HL161385"],"award-info":[{"award-number":["R01HL161385"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Large language models (LLMs) have shown incredible promise in medicine. While LLMs may be particularly useful in areas requiring extensive review of clinical records, their use remains limited due to their tendency to hallucinate and fabricate information. Hallucination issues, as well as their consequences, are exacerbated in low\u2013probability, high\u2013stakes scenarios such as rare adverse safety events or medical errors. We present SAFE\u2013AI (Structured and Automated Framework for Explainable AI), a novel method for clinical decision making that combines the strengths of clinical expert knowledge with LLMs in an ontology\u2013driven model that minimizes hallucinations using strict rules. We test this method to identify medication errors in medical charts. We collected a sample of 18,402 lines of clinical information from 300 EMS clinical charts that were independently dually reviewed by two expert physicians for epinephrine adverse safety events (ASEs), with 96% inter-rater agreement. We tested SAFE\u2013AI against these labels, achieving similar performance to human experts in detecting epinephrine overdoses with 97.9% accuracy, and 91.6% accuracy in identifying delays in epinephrine administration, greatly outperforming baseline LLMs models. Notably, some disagreements between clinicians and the model were found to be justifiable differences in judgment rather than errors. SAFE-AI presents a novel approach for clinical AI applications that addresses two key limitations of current machine learning methods: 1) over-reliance on probabilistic pattern recognition instead of established medical knowledge, and 2) perpetuation of biases present in training data. This framework is easily adaptable to a range of clinical applications, paving the way for provable and trustworthy AI in medicine.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001174","type":"journal-article","created":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T17:26:25Z","timestamp":1789665985000},"page":"e0001174","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["A medically grounded LLM agent\u2013based tool to detect patient safety events in medical records"],"prefix":"10.1371","volume":"5","author":[{"given":"Diego","family":"Trujillo","sequence":"first","affiliation":[{"name":"Center for Learning Health Care Delivery, Beth Israel Deaconess Medical Center","place":["Boston, Massachusetts, United States of America"]},{"name":"Beth Israel Deaconess Medical Center and Harvard Medical School","place":["Boston, Massachusetts, United States of America"],"department":["Department of Obstetrics Gynecology and Reproductive Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dulin","family":"Wang","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, UTHealth","place":["Houston, Texas, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan","family":"Bahr","sequence":"additional","affiliation":[{"name":"Oregon Health and Science University","place":["Portland, Oregon, United States of America"],"department":["Department of Emergency Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6296-0689","authenticated-orcid":true,"given":"Tina","family":"Yi-Jin Hsieh","sequence":"additional","affiliation":[{"name":"Beth Israel Deaconess Medical Center and Harvard Medical School","place":["Boston, Massachusetts, United States of America"],"department":["Department of Obstetrics Gynecology and Reproductive Medicine"]},{"name":"Harvard Medical School","place":["Boston, Massachusetts, United States of America"],"department":["Department of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1370-2629","authenticated-orcid":true,"given":"Byeongyeon","family":"Cho","sequence":"additional","affiliation":[{"name":"Center for Learning Health Care Delivery, Beth Israel Deaconess Medical Center","place":["Boston, Massachusetts, United States of America"]},{"name":"Harvard Medical School","place":["Boston, Massachusetts, United States of America"],"department":["Department of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Garth","family":"Meckler","sequence":"additional","affiliation":[{"name":"University of British Columbia","place":["Vancouver, Canada"],"department":["Department of Pediatrics and Emergency Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Hansen","sequence":"additional","affiliation":[{"name":"Oregon Health and Science University","place":["Portland, Oregon, United States of America"],"department":["Department of Emergency Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carl","family":"Eriksson","sequence":"additional","affiliation":[{"name":"Oregon Health and Science University","place":["Portland, Oregon, United States of America"],"department":["Department of Pediatrics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyu","family":"Seo Kim","sequence":"additional","affiliation":[{"name":"Oregon Health and Science University","place":["Portland, Oregon, United States of America"],"department":["Department of Medical Informatics and Clinical Epidemiology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven","family":"Bedrick","sequence":"additional","affiliation":[{"name":"Oregon Health and Science University","place":["Portland, Oregon, United States of America"],"department":["Department of Medical Informatics and Clinical Epidemiology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9933-2205","authenticated-orcid":true,"given":"Xiaoqian","family":"Jiang","sequence":"additional","affiliation":[{"name":"McWilliams School of Biomedical Informatics, UTHealth","place":["Houston, Texas, United States of America"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8961-488X","authenticated-orcid":true,"given":"Jeanne-Marie","family":"Guise","sequence":"additional","affiliation":[{"name":"Center for Learning Health Care Delivery, Beth Israel Deaconess Medical Center","place":["Boston, Massachusetts, United States of America"]},{"name":"Beth Israel Deaconess Medical Center and Harvard Medical School","place":["Boston, Massachusetts, United States of America"],"department":["Department of Obstetrics Gynecology and Reproductive Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,17]]},"reference":[{"key":"pdig.0001174.ref001","first-page":"13","article-title":"Your health care may kill you: medical errors","volume-title":"Building Capacity for Health Informatics in the Future","author":"JG Anderson","year":"2017"},{"issue":"6","key":"pdig.0001174.ref002","doi-asserted-by":"crossref","first-page":"530","DOI":"10.7861\/clinmedicine.16-6-530","article-title":"Learning from hospital mortality","volume":"16","author":"K Stewart","year":"2016","journal-title":"Clin Med (Lond)"},{"key":"pdig.0001174.ref003","unstructured":"Donaldson MS, Corrigan JM, Kohn LT. To err is human: building a safer health system. 2000."},{"key":"pdig.0001174.ref004","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1016\/j.jbi.2003.08.003","article-title":"Detecting adverse events for patient safety research: a review of current methodologies","volume":"36","author":"HJ Murff","year":"2003","journal-title":"Journal of Biomedical Informatics"},{"key":"pdig.0001174.ref005","unstructured":"Rosen AK. Are We Getting Better at Measuring Patient Safety? PSNet. 2010."},{"key":"pdig.0001174.ref006","article-title":"Comparing rates of adverse events detected in incident reporting and the Global Trigger Tool: a systematic review","volume":"35","author":"PD Hibbert","journal-title":"International Journal for Quality in Health Care 2Jul"},{"key":"pdig.0001174.ref007","unstructured":"Xu Z, Jain S, Kankanhalli MS. Hallucination is inevitable: an innate limitation of large language models. ArXiv. 2024. https:\/\/api.semanticscholar.org\/CorpusID:267069207"},{"issue":"5","key":"pdig.0001174.ref008","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1016\/j.cgh.2023.09.035","article-title":"Evaluating ChatGPT in Medical Contexts: The Imperative to Guard Against Hallucinations and Partial Accuracies","volume":"22","author":"M Giuffr\u00e8","year":"2024","journal-title":"Clin Gastroenterol Hepatol"},{"key":"pdig.0001174.ref009","doi-asserted-by":"crossref","first-page":"1495582","DOI":"10.3389\/fmed.2024.1495582","article-title":"Why we need to be careful with LLMs in medicine","volume":"11","author":"JC Be\u00b4lisle-Pipon","year":"2024","journal-title":"Frontiers in Medicine"},{"key":"pdig.0001174.ref010","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2322420121","article-title":"Embers of autoregression show how large language models are shaped by the problem they are trained to solve","volume":"121","author":"RT McCoy","year":"2024","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pdig.0001174.ref011","unstructured":"Mirzadeh I, Alizadeh K, Shahrokhi H, Tuzel O, Bengio S, Farajtabar M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. 2024. https:\/\/arxiv.org\/abs\/2410.05229"},{"key":"pdig.0001174.ref012","unstructured":"Gao Y, Xiong Y, Gao X, Jia K, Pan J, Bi Y, et al. Retrieval-Augmented Generation for Large Language Models: A Survey. 2023. https:\/\/api.semanticscholar.org\/CorpusID:266359151"},{"key":"pdig.0001174.ref013","unstructured":"Yang X, Sun K, Xin H, Sun Y, Bhalla N, Chen X. CRAG \u2013 Comprehensive RAG Benchmark. In: 2024. https:\/\/arxiv.org\/abs\/2406.04744"},{"issue":"1","key":"pdig.0001174.ref014","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2023.51535","article-title":"Adverse Safety Events in Emergency Medical Services Care of Children With Out-of-Hospital Cardiac Arrest","volume":"7","author":"CO Eriksson","year":"2024","journal-title":"JAMA Netw Open"},{"key":"pdig.0001174.ref015","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1080\/10903120600724200","article-title":"National emergency medical services information system (NEMSIS)","volume":"10","author":"DE Dawson","year":"2006","journal-title":"Prehospital Emergency Care"},{"issue":"10","key":"pdig.0001174.ref016","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2020-039215","article-title":"Retrospective chart review and survey to identify adverse safety events in the emergency medical services care of children with out-of-hospital cardiac arrest in the USA: a study protocol","volume":"10","author":"C Eriksson","year":"2020","journal-title":"BMJ Open"},{"key":"pdig.0001174.ref017","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1016\/j.ajem.2017.08.028","article-title":"Safety events in pediatric out-of-hospital cardiac arrest","volume":"36","author":"M Hansen","year":"2018","journal-title":"The American Journal of Emergency Medicine"},{"key":"pdig.0001174.ref018","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"KL Gwet","year":"2008","journal-title":"Br J Math Stat Psychol"},{"issue":"3","key":"pdig.0001174.ref019","doi-asserted-by":"crossref","DOI":"10.1002\/emp2.12726","article-title":"A chart review tool to systematically assess the safety of prehospital care for children with out\u2010of\u2010hospital cardiac arrest","volume":"3","author":"A Schoonover","year":"2022","journal-title":"JACEP Open"},{"issue":"6","key":"pdig.0001174.ref020","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1016\/0895-4356(90)90159-M","article-title":"High agreement but low kappa: II. Resolving the paradoxes","volume":"43","author":"DV Cicchetti","year":"1990","journal-title":"J Clin Epidemiol"},{"key":"pdig.0001174.ref021","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples","volume":"13","author":"N Wongpakaran","year":"2013","journal-title":"BMC Med Res Methodol"},{"key":"pdig.0001174.ref022","doi-asserted-by":"crossref","first-page":"211","DOI":"10.2174\/1874434601711010211","article-title":"High Agreement and High Prevalence: The Paradox of Cohen\u2019s Kappa","volume":"11","author":"S Zec","year":"2017","journal-title":"Open Nurs J"},{"key":"pdig.0001174.ref023","unstructured":"Hegselmann S, Buendia A, Lang H, Agrawal M, Jiang X, Sontag D. TabLLM: Few-shot Classification of Tabular Data with Large Language Models. In: Proceedings of Machine Learning Research, 2023. 5549\u201381. https:\/\/proceedings.mlr.press\/v206\/hegselmann23a.html"},{"key":"pdig.0001174.ref024","volume-title":"Natural language processing with Python: analyzing text with the natural language toolkit","author":"S Bird","year":"2009"},{"key":"pdig.0001174.ref025","doi-asserted-by":"crossref","DOI":"10.1161\/CIR.0000000000000901","article-title":"Part 4: Pediatric Basic and Advanced Life Support: 2020 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care","volume":"142","author":"AA Topjian","year":"2020","journal-title":"Circulation"},{"key":"pdig.0001174.ref026","article-title":"Medical error\u2014the third leading cause of death in the US","author":"MA Makary","year":"2016","journal-title":"BMJ"},{"issue":"6","key":"pdig.0001174.ref027","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1056\/NEJM199102073240604","article-title":"Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I","volume":"324","author":"TA Brennan","year":"1991","journal-title":"N Engl J Med"},{"key":"pdig.0001174.ref028","first-page":"255","article-title":"Costs of medical injuries in Utah and Colorado","author":"EJ Thomas","year":"1999","journal-title":"Inquiry"},{"issue":"5","key":"pdig.0001174.ref029","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/S0097-5990(16)30608-X","article-title":"Preventing medical injury","volume":"19","author":"LL Leape","year":"1993","journal-title":"QRB Qual Rev Bull"},{"issue":"23","key":"pdig.0001174.ref030","doi-asserted-by":"crossref","first-page":"1851","DOI":"10.1001\/jama.1994.03520230061039","article-title":"Error in medicine","volume":"272","author":"LL Leape","year":"1994","journal-title":"JAMA"},{"key":"pdig.0001174.ref031","unstructured":"HealthGrades. HealthGrades Quality Study: Patient Safety in American Hospitals. 2004. http:\/\/www.providersedge.com\/ehdocs\/ehr_articles\/Patient_Safety_in_American_Hospitals-2004.pdf"},{"key":"pdig.0001174.ref032","volume-title":"Adverse events in hospitals: national incidence among Medicare beneficiaries","author":"DR Levinson","year":"2010"},{"issue":"1","key":"pdig.0001174.ref033","first-page":"39","article-title":"Defining medical error","volume":"48","author":"ED Grober","year":"2005","journal-title":"Can J Surg"},{"key":"pdig.0001174.ref034","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1001\/jama.2015.9678","article-title":"Time to epinephrine and survival after pediatric in-hospital cardiac arrest","volume":"314","author":"LW Andersen","year":"2015","journal-title":"JAMA"},{"issue":"3","key":"pdig.0001174.ref035","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1097\/SIH.0000000000000013","article-title":"Errors and error-producing conditions during a simulated, prehospital, pediatric cardiopulmonary arrest","volume":"9","author":"RL Lammers","year":"2014","journal-title":"Simul Healthc"},{"issue":"1","key":"pdig.0001174.ref036","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1080\/10903127.2022.2162648","article-title":"An Analysis of Prehospital Pediatric Medication Dosing Errors after Implementation of a State-Wide EMS Pediatric Drug Dosing Reference","volume":"28","author":"R Kazi","year":"2024","journal-title":"Prehosp Emerg Care"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001174","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T17:26:36Z","timestamp":1789665996000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001174"}},"subtitle":[],"editor":[{"given":"Xiaoli","family":"Liu","sequence":"first","affiliation":[{"name":"Chinese PLA General Hospital","place":["CHINA"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,17]]},"references-count":36,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,17]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001174","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,17]]}}}