{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T18:02:33Z","timestamp":1790359353308,"version":"4.1.0"},"reference-count":56,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T00:00:00Z","timestamp":1790294400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>\n                    We investigated the potential utility of large language models (LLMs) in supporting patient safety efforts. Specifically, we evaluated the reasoning capabilities of LLMs in performing root cause analysis (RCA) of radiation oncology incidents using narrative reports from the Radiation Oncology Incident Learning System (RO-ILS). We prompted four state-of-the-art LLMs, Gemini 2.5 Pro, GPT-4o, o3, and Grok 3, with the \u201cBackground and Incident Overview\u201d sections from 19 publicly available RO-ILS cases. Each model was instructed to perform RCA and generate root causes, lessons learned, and suggested actions using a standardized prompt based on AAPM RCA guidelines. Model outputs were evaluated using a combination of objective semantic similarity metrics (cosine similarity via Sentence Transformer), semi-subjective assessments (precision, recall, F1-score, expert-adjudicated PPV (Positive Predictive Value), hallucination rate and performance criteria including relevance, comprehensiveness, quality of justification and quality of solution), and subjective ratings (reasoning quality and overall performance) by five board-certified medical physicists. LLMs demonstrated satisfactory performance across evaluation metrics. All models exhibited some degree of hallucination, ranging from 11% to 61%. All the evaluated LLMs demonstrated comparable baseline capabilities in objective causal extraction, and Gemini 2.5 Pro exhibited the highest overall performance score among 4 models. Statistically significant differences were observed among models in expert-adjudicated PPV, hallucination rate, and subjective ratings (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.05). LLMs delivered promising results as assistive tools for RCA in radiation oncology, with the ability to generate relevant and accurate analyses aligned with expert expectations. LLMs may support incident analysis and contribute to quality improvement efforts to advance patient safety in clinical radiation oncology practice.\n                  <\/jats:p>","DOI":"10.1371\/journal.pdig.0001740","type":"journal-article","created":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T17:43:52Z","timestamp":1790358232000},"page":"e0001740","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Augmenting patient safety surveillance in radiation oncology with large language model-based root cause analysis: A proof-of-concept study"],"prefix":"10.1371","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-5499-6258","authenticated-orcid":true,"given":"Yuntao","family":"Wang","sequence":"first","affiliation":[{"name":"University of Miami","place":["Miami, Florida, United States of America"],"department":["Department of Radiation Oncology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mariluz","family":"De Ornelas","sequence":"additional","affiliation":[{"name":"University of Miami","place":["Miami, Florida, United States of America"],"department":["Department of Radiation Oncology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew T.","family":"Studenski","sequence":"additional","affiliation":[{"name":"University of Miami","place":["Miami, Florida, United States of America"],"department":["Department of Radiation Oncology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth","family":"Bossart","sequence":"additional","affiliation":[{"name":"University of Miami","place":["Miami, Florida, United States of America"],"department":["Department of Radiation Oncology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Siamak P.","family":"Nejad-Davarani","sequence":"additional","affiliation":[{"name":"University of Miami","place":["Miami, Florida, United States of America"],"department":["Department of Radiation Oncology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4776-4588","authenticated-orcid":true,"given":"Yunze","family":"Yang","sequence":"additional","affiliation":[{"name":"University of Miami","place":["Miami, Florida, United States of America"],"department":["Department of Radiation Oncology"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,25]]},"reference":[{"issue":"10","key":"pdig.0001740.ref001","doi-asserted-by":"crossref","first-page":"3270","DOI":"10.1038\/s41591-025-03983-2","article-title":"Generative artificial intelligence in medicine","volume":"31","author":"ZL Teo","year":"2025","journal-title":"Nat Med"},{"issue":"1","key":"pdig.0001740.ref002","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"B Mesk\u00f3","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"1","key":"pdig.0001740.ref003","doi-asserted-by":"crossref","first-page":"450","DOI":"10.1038\/s41746-025-01824-7","article-title":"Large language model integrations in cancer decision-making: a systematic review and meta-analysis","volume":"8","author":"Y Hao","year":"2025","journal-title":"NPJ Digit Med"},{"key":"pdig.0001740.ref004","article-title":"Reasoning with large language models, a survey","author":"A Plaat","year":"2024"},{"key":"pdig.0001740.ref005","article-title":"Towards reasoning in large language models: A survey. arXiv preprint","author":"J Huang","year":"2022"},{"key":"pdig.0001740.ref006","article-title":"From system 1 to system 2: A survey of reasoning large language models. arXiv preprint","author":"ZZ Li","year":"2025"},{"issue":"120","key":"pdig.0001740.ref007","first-page":"1","article-title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","volume":"23","author":"W Fedus","year":"2022","journal-title":"Journal of Machine Learning Research"},{"key":"pdig.0001740.ref008","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"T Brown","year":"2020","journal-title":"Advances in neural information processing systems"},{"key":"pdig.0001740.ref009","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TKDE.2025.3554028","article-title":"A Survey on Mixture of Experts in Large Language Models","author":"W Cai","year":"2025","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"pdig.0001740.ref010","article-title":"Advancing reasoning in large language models: promising methods and approaches","author":"A Patil","year":"2025"},{"key":"pdig.0001740.ref011","article-title":"Supervised contrastive learning for pre-trained language model fine-tuning. arXiv preprint","author":"B Gunel","year":"2020"},{"key":"pdig.0001740.ref012","article-title":"Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint","author":"Y Bai","year":"2022"},{"key":"pdig.0001740.ref013","article-title":"Towards large reasoning models: A survey of reinforced reasoning with large language models","author":"F Xu","year":"2025"},{"key":"pdig.0001740.ref014","doi-asserted-by":"crossref","unstructured":"Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, et al. Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems 35, 2022. 24824\u201337. https:\/\/doi.org\/10.52202\/068431-1800","DOI":"10.52202\/068431-1800"},{"key":"pdig.0001740.ref015","doi-asserted-by":"crossref","unstructured":"Yao S, Yu D, Zhao J, Shafran I, Griffiths T, Cao Y, et al. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In: Advances in Neural Information Processing Systems 36, 2023. 11809\u201322. https:\/\/doi.org\/10.52202\/075280-0517","DOI":"10.52202\/075280-0517"},{"key":"pdig.0001740.ref016","doi-asserted-by":"crossref","unstructured":"Kojima T, Gu S (Shane), Reid M, Matsuo Y, Iwasawa Y. Large Language Models Are Zero-Shot Reasoners. In: Advances in Neural Information Processing Systems 35, 2022. 22199\u2013213. https:\/\/doi.org\/10.52202\/068431-1613","DOI":"10.52202\/068431-1613"},{"key":"pdig.0001740.ref017","article-title":"Beyond the imitation game: quantifying and extrapolating the capabilities of language models","author":"A Srivastava","year":"2023","journal-title":"Transactions on Machine Learning Research"},{"key":"pdig.0001740.ref018","article-title":"Evaluating large language models trained on code","author":"M Chen","year":"2021"},{"key":"pdig.0001740.ref019","unstructured":"Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D. Measuring massive multitask language understanding. In: 2020. https:\/\/arxiv.org\/abs\/2009.03300"},{"key":"pdig.0001740.ref020","unstructured":"Cobbe K, Kosaraju V, Bavarian M, Chen M, Jun H, Kaiser L, et al. Training verifiers to solve math word problems. In: 2021. https:\/\/arxiv.org\/abs\/2110.14168"},{"key":"pdig.0001740.ref021","unstructured":"Rein D, Hou BL, Stickland AC, Petty J, Pang RY, Dirani J. Gpqa: A graduate-level google-proof q&a benchmark. In: 2024."},{"key":"pdig.0001740.ref022","unstructured":"Huang Y, Yang LF. Gemini 2.5 Pro capable of winning gold at IMO 2025. In: 2025. https:\/\/doi.org\/arXiv:250715855"},{"key":"pdig.0001740.ref023","doi-asserted-by":"crossref","unstructured":"Ferrag MA, Tihanyi N, Debbah M. From llm reasoning to autonomous ai agents: A comprehensive review. 2025. https:\/\/arxiv.org\/abs\/250419678","DOI":"10.1109\/ACCESS.2026.3698694"},{"issue":"2","key":"pdig.0001740.ref024","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3703155","article-title":"A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions","volume":"43","author":"L Huang","year":"2025","journal-title":"ACM Trans Inf Syst"},{"issue":"1","key":"pdig.0001740.ref025","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1017\/S1460396902000122","article-title":"Preventing treatment errors in radiotherapy by identifying and evaluating near misses and actual incidents","volume":"3","author":"O Holmberg","year":"2002","journal-title":"J Radiother Pract"},{"key":"pdig.0001740.ref026","volume-title":"Understanding patient safety","author":"RM Wachter","year":"2008"},{"issue":"5","key":"pdig.0001740.ref027","article-title":"Incident learning in radiation oncology: A review","volume":"45","author":"EC Ford","year":"2018","journal-title":"Med Phys"},{"key":"pdig.0001740.ref028","unstructured":"Evans SB, Viscariello NN. Learning from ten-years of RO-Ils data: how to avoid common error pathways and engage the entire team. In: 2025."},{"issue":"5","key":"pdig.0001740.ref029","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1016\/j.prro.2015.06.009","article-title":"RO-ILS: Radiation Oncology Incident Learning System: A report from the first year of experience","volume":"5","author":"DJ Hoopes","year":"2015","journal-title":"Pract Radiat Oncol"},{"key":"pdig.0001740.ref030","doi-asserted-by":"crossref","DOI":"10.1080\/08998280.2001.11927753","article-title":"Techniques for root cause analysis","volume-title":"Baylor University Medical Center Proceedings","author":"PM Williams","year":"2001"},{"key":"pdig.0001740.ref031","doi-asserted-by":"crossref","DOI":"10.1201\/9781420087284","volume-title":"Patient safety: the PROACT root cause analysis approach","author":"RJ Latino","year":"2008"},{"key":"pdig.0001740.ref032","unstructured":"RO-ILS Education. https:\/\/www.astro.org\/practice-support\/quality-and-safety\/ro-ils\/ro-ils-education"},{"key":"pdig.0001740.ref033","unstructured":"Team G, Anil R, Borgeaud S, Alayrac JB, Yu J, Soricut R. Gemini: a family of highly capable multimodal models. 2023. https:\/\/arxiv.org\/abs\/2312.11805"},{"key":"pdig.0001740.ref034","unstructured":"Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, et al. Gpt-4 technical report. 2023. https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"pdig.0001740.ref035","first-page":"16857","article-title":"Mpnet: Masked and permuted pre-training for language understanding","volume":"33","author":"K Song","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"pdig.0001740.ref036","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"KL Gwet","year":"2008","journal-title":"Br J Math Stat Psychol"},{"issue":"7972","key":"pdig.0001740.ref037","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"K Singhal","year":"2023","journal-title":"Nature"},{"issue":"3","key":"pdig.0001740.ref038","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"K Singhal","year":"2025","journal-title":"Nat Med"},{"issue":"9","key":"pdig.0001740.ref039","doi-asserted-by":"crossref","first-page":"1964","DOI":"10.1093\/jamia\/ocae131","article-title":"Reasoning with large language models for medical question answering","volume":"31","author":"MM Lucas","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"pdig.0001740.ref040","doi-asserted-by":"crossref","first-page":"100153","DOI":"10.1016\/j.metrad.2025.100153","article-title":"Radiology-GPT: A large language model for radiology","volume":"3","author":"Z Liu","year":"2025","journal-title":"Meta-Radiology"},{"issue":"1","key":"pdig.0001740.ref041","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1016\/j.ijrobp.2024.07.2149","article-title":"Autodelineation of Treatment Target Volume for Radiation Therapy Using Large Language Model-Aided Multimodal Learning","volume":"121","author":"P Rajendran","year":"2025","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"pdig.0001740.ref042","doi-asserted-by":"crossref","first-page":"110740","DOI":"10.1016\/j.radonc.2025.110740","article-title":"Large language model-augmented learning for auto-delineation of treatment targets in head-and-neck cancer radiotherapy","volume":"205","author":"P Rajendran","year":"2025","journal-title":"Radiother Oncol"},{"issue":"1","key":"pdig.0001740.ref043","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1002\/mp.17416","article-title":"Large-language-model empowered 3D dose prediction for intensity-modulated radiotherapy","volume":"52","author":"Z Dong","year":"2025","journal-title":"Med Phys"},{"issue":"1","key":"pdig.0001740.ref044","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1186\/s13014-025-02660-5","article-title":"Feasibility study of automatic radiotherapy treatment planning for cervical cancer using a large language model","volume":"20","author":"S Wei","year":"2025","journal-title":"Radiat Oncol"},{"issue":"7","key":"pdig.0001740.ref045","doi-asserted-by":"crossref","DOI":"10.1002\/mp.17985","article-title":"Fine-tuning open-source large language models to improve their performance on radiation oncology tasks: A feasibility study to investigate their potential clinical applications in radiation oncology","volume":"52","author":"P Wang","year":"2025","journal-title":"Med Phys"},{"issue":"4","key":"pdig.0001740.ref046","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2024.4630","article-title":"Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions","volume":"7","author":"A Yalamanchili","year":"2024","journal-title":"JAMA Netw Open"},{"issue":"1","key":"pdig.0001740.ref047","doi-asserted-by":"crossref","first-page":"100198","DOI":"10.1016\/j.mcpdig.2025.100198","article-title":"Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses From Closed-Domain Large Language Models Versus Clinical Teams","volume":"3","author":"Y Hao","year":"2025","journal-title":"Mayo Clin Proc Digit Health"},{"key":"pdig.0001740.ref048","doi-asserted-by":"crossref","first-page":"1219326","DOI":"10.3389\/fonc.2023.1219326","article-title":"Evaluating large language models on a highly-specialized topic, radiation oncology physics","volume":"13","author":"J Holmes","year":"2023","journal-title":"Front Oncol"},{"issue":"8","key":"pdig.0001740.ref049","doi-asserted-by":"crossref","DOI":"10.1002\/mp.17936","article-title":"Semi-automated topic identification for radiation oncology safety event reports using natural language processing and statistical modeling","volume":"52","author":"QS Zhang","year":"2025","journal-title":"Med Phys"},{"key":"pdig.0001740.ref050","unstructured":"Beidler P, Nguyen M, Lybarger K, Holmberg O, Ford E, Kang J. Automated Triaging and Transfer Learning of Incident Learning Safety Reports Using Large Language Representational Models. arXiv preprint arXiv:250913706. 2025."},{"key":"pdig.0001740.ref051","article-title":"Large language models in cardiovascular imaging: current applications and future prospects","author":"W Yuan","year":"2025","journal-title":"Med Research"},{"issue":"3","key":"pdig.0001740.ref052","first-page":"241","article-title":"miR-634 Decreases the Radioresistance of Human Breast Cancer Cells by Targeting STAT3","volume":"35","author":"B Yang","year":"2020","journal-title":"Cancer Biother Radiopharm"},{"key":"pdig.0001740.ref053","doi-asserted-by":"crossref","first-page":"239","DOI":"10.2147\/IJN.S287523","article-title":"Au-Pt Nanoparticle Formulation as a Radiosensitizer for Radiotherapy with Dual Effects","volume":"16","author":"S Yang","year":"2021","journal-title":"Int J Nanomedicine"},{"issue":"9","key":"pdig.0001740.ref054","doi-asserted-by":"crossref","first-page":"5858","DOI":"10.1002\/mp.17282","article-title":"AAPM task group report 288: Recommendations for guiding radiotherapy event narratives","volume":"51","author":"B Thomadsen","year":"2024","journal-title":"Medical Physics"},{"issue":"7","key":"pdig.0001740.ref055","doi-asserted-by":"crossref","first-page":"4209","DOI":"10.1118\/1.4947547","article-title":"The report of Task Group 100 of the AAPM: Application of risk analysis methods to radiation therapy quality management","volume":"43","author":"MS Huq","year":"2016","journal-title":"Medical Physics"},{"key":"pdig.0001740.ref056","article-title":"Quantitative Risk Assessment in Radiation Oncology via LLM-Powered Root Cause Analysis of Incident Reports","author":"Y Wang","year":"2025"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001740","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T17:44:02Z","timestamp":1790358242000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001740"}},"subtitle":[],"editor":[{"given":"Javier","family":"Parapar","sequence":"first","affiliation":[{"name":"University of A Coruna: Universidade da Coruna","place":["SPAIN"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,25]]},"references-count":56,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,25]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001740","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,25]]}}}