{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T04:45:43Z","timestamp":1791261943137,"version":"4.1.0"},"reference-count":36,"publisher":"American Society of Clinical Oncology (ASCO)","issue":"9","content-domain":{"domain":["ascopubs.org"],"crossmark-restriction":true},"short-container-title":["JCO Clin Cancer Inform"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:sec>\n                    <jats:title>PURPOSE<\/jats:title>\n                    <jats:p>Large language models (LLMs) show promise in assisting knowledge-intensive fields such as oncology, where up-to-date information and multidisciplinary expertise are critical. Traditional LLMs risk hallucinations and reliance on static, possibly outdated data that lack domain-specific context. Retrieval-augmented generation (RAG) has emerged as a strategy to address these issues by incorporating domain-specific information from external knowledge repositories.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>METHODS<\/jats:title>\n                    <jats:p>We evaluated 15 LLMs, including Meta Llama-2\/3, generative pretrained transformer (GPT)-3.5\/4\/4o variants, Claude-3, Gemini-2.0, and DeepSeek-R1. In a zero-shot workflow, each LLM answered 298 scorable questions from the 2021 American College of Radiology in-training examination. We implemented a RAG pipeline (Iridium Model) that transforms user prompts into vector embeddings, queries a specialized radiation oncology database, and merges relevant text with the original prompt to form an augmented query. We compared zero-shot versus RAG-augmented performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>RESULTS<\/jats:title>\n                    <jats:p>\n                      Larger-parameter LLMs had higher zero-shot accuracy, with six models outscoring graduating residents (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      &lt; .01). Top scorers were reasoning models GPT-4o1, o3-mini, and DeepSeek-R1, which achieved 91.6%, 86.6%, and 91.6% without RAG, respectively. Gemini-2.0 improved 6.7% (to 79.2%), Llama-3-70b 8.4% (to 75.8%), and GPT-4o 5.7% (to 85.6%) with RAG. Top scoring reasoning models surpassed graduating resident averages by 17.7%-20% (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      &lt; .01), but had no improvement or detriment with RAG. Domain-specific gains occurred in clinical, biology, and physics. Majority voting boosted aggregate accuracy when individual model performance exceeded 50%. RAG workflows and reasoning models incurred higher computational costs.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>CONCLUSION<\/jats:title>\n                    <jats:p>Radiation-oncology\u2013specific retrieval-augmented generation pipeline enhances nonreasoning LLM performance in radiation oncology by integrating domain-specific evidence, whereas it does not improve performance of reasoning models. These findings demonstrate that RAG can elevate clinical decision support by enabling simpler, cost-effective nonreasoning models to tackle complex tasks through retrieval capabilities\u2014an efficient alternative to extensive model training that also yields citable, evidence-based explanations.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1200\/cci-25-00220","type":"journal-article","created":{"date-parts":[[2025,11,14]],"date-time":"2025-11-14T21:00:22Z","timestamp":1763154022000},"update-policy":"https:\/\/doi.org\/10.1200\/crossmark","source":"Crossref","is-referenced-by-count":7,"title":["RadOncRAG: A Novel Retrieval-Augmented Generation Framework Improves Large Language Model Benchmark Performance in Radiation Oncology"],"prefix":"10.1200","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3608-0025","authenticated-orcid":true,"given":"Nikhil Gautam","family":"Thaker","sequence":"first","affiliation":[{"name":"Capital Health, Pennington, NJ"},{"name":"Bayta Systems, Newtown, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0951-2195","authenticated-orcid":true,"given":"Navid","family":"Redjal","sequence":"additional","affiliation":[{"name":"Capital Health, Pennington, NJ"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0733-3337","authenticated-orcid":true,"given":"Adam","family":"Dicker","sequence":"additional","affiliation":[{"name":"Thomas Jefferson, Philadelphia, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3655-9602","authenticated-orcid":true,"given":"Arturo","family":"Loaiza-Bonilla","sequence":"additional","affiliation":[{"name":"St Luke's University Health Network, Easton, PA"},{"name":"Massive Bio, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9879-1290","authenticated-orcid":true,"given":"Trevor","family":"Royce","sequence":"additional","affiliation":[{"name":"Atrium Health Wake Forest Baptist, Winston Salem, NC"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6064-6837","authenticated-orcid":true,"given":"Vivek","family":"Subbiah","sequence":"additional","affiliation":[{"name":"Sarah Cannon Research Institute, Nashville, TN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vikash","family":"Deendyal","sequence":"additional","affiliation":[{"name":"Capital Health, Pennington, NJ"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan R.","family":"Gabriel","sequence":"additional","affiliation":[{"name":"University of Arizona, Tucson, AZ"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neena","family":"Shetty","sequence":"additional","affiliation":[{"name":"Capital Health, Pennington, NJ"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ajay","family":"Choudhri","sequence":"additional","affiliation":[{"name":"Capital Health, Pennington, NJ"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gautam H.","family":"Thaker","sequence":"additional","affiliation":[{"name":"Bayta Systems, Newtown, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"233","reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iotcps.2023.04.003"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02366-9"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1002\/1878-0261.12712"},{"key":"e_1_3_4_5_2","first-page":"499","article-title":"Growth and decentralization of the medical literature: Implications for evidence-based medicine","volume":"93","author":"Druss BG","year":"2005","unstructured":"Druss BG, Marcus SC: Growth and decentralization of the medical literature: Implications for evidence-based medicine. J Med Libr Assoc 93:499-501, 2005","journal-title":"J Med Libr Assoc"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1162\/qss_a_00327"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-022-01984-7"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.adro.2023.101206"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-019-0217-0"},{"key":"e_1_3_4_10_2","first-page":"e35179","article-title":"Artificial hallucinations in ChatGPT: Implications in scientific writing","volume":"15","author":"Alkaissi H","year":"2023","unstructured":"Alkaissi H, McFarlane SI: Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus 15:e35179, 2023","journal-title":"Cureus"},{"key":"e_1_3_4_11_2","article-title":"RadOnc-GPT: A large language model for radiation oncology","volume":"9","author":"Liu Z","year":"2023","unstructured":"Liu Z, Wang P, Li Y, et al: RadOnc-GPT: A large language model for radiation oncology. ArXiv Med Phys 9, 2023","journal-title":"ArXiv Med Phys"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1089\/aipo.2023.0007"},{"key":"e_1_3_4_13_2","doi-asserted-by":"crossref","first-page":"e134","DOI":"10.1016\/j.ijrobp.2024.07.2078","article-title":"RadAIonc: Enhancing decision-making in radiation oncology with a GPT-4-based prompt-driven large language model","volume":"120","author":"Han B","year":"2024","unstructured":"Han B, Chen Y, Buyyounouski MK, et al: RadAIonc: Enhancing decision-making in radiation oncology with a GPT-4-based prompt-driven large language model. Int J Radiat Oncol Biol Phys 120:e134, 2024 (suppl 2)","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1056\/AIra2400380"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.3390\/medicina60030445"},{"key":"e_1_3_4_16_2","volume-title":"AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed Articles","author":"Lahiri A","year":"2024","unstructured":"Lahiri A, Hu Q: AlzheimerRAG: Multimodal Retrieval Augmented Generation for PubMed Articles. 2024"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01391-3"},{"key":"e_1_3_4_18_2","unstructured":"ASTRO: Guidelines - American Society for Radiation Oncology (ASTRO). 2025. https:\/\/www.astro.org\/provider-resources\/guidelines\/clinical-practice-guidelines"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacr.2008.04.006"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacr.2020.04.032"},{"key":"e_1_3_4_21_2","unstructured":"Thaker N: Thakernikhil\/iridium-radonc-rag [Python]. 2025. https:\/\/github.com\/thakernikhil\/iridium-radonc-rag"},{"key":"e_1_3_4_22_2","volume-title":"Majority Voting and the Condorcet's Jury Theorem","author":"Shteingart H","year":"2020","unstructured":"Shteingart H, Marom E, Itkin I, et al: Majority Voting and the Condorcet's Jury Theorem. 2020"},{"key":"e_1_3_4_23_2","volume-title":"American College of Radiology. Radiation Oncology In-Training Examination. National Statistical Summary. Administered in March 2021","author":"ACR","year":"2021","unstructured":"ACR: American College of Radiology. Radiation Oncology In-Training Examination. National Statistical Summary. Administered in March 2021. 2021"},{"key":"e_1_3_4_24_2","volume-title":"Hyperbolic\u2014The Open-Access AI Cloud","author":"Anonymous","unstructured":"Anonymous: Hyperbolic\u2014The Open-Access AI Cloud. n.d. https:\/\/hyperbolic.xyz\/"},{"key":"e_1_3_4_25_2","volume-title":"Summarization of Large Text Volumes Using Large Language Models","author":"Betri\u00e1n ADN","year":"2024","unstructured":"Betri\u00e1n ADN, Kaltenbrunner A: Summarization of Large Text Volumes Using Large Language Models. 2024"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","unstructured":"Zhou J Lu T Mishra S et al: Instruction-following evaluation for large language models. 2023. 10.48550\/arXiv.2311.07911","DOI":"10.48550\/arXiv.2311.07911"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01190-w"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","unstructured":"Su C Wen J Kang J et al: Hybrid RAG-empowered multi-modal LLM for secure data management in internet of medical things: A diffusion-based contract approach. 2024. 10.48550\/arXiv.2407.00978","DOI":"10.48550\/arXiv.2407.00978"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-024-01368-4"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104564"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-23901-7"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.1838"},{"key":"e_1_3_4_33_2","volume-title":"GPT-4o vs DeepSeek-V3\u2014Detailed Performance & Feature Comparison","author":"Anonymous","unstructured":"Anonymous: GPT-4o vs DeepSeek-V3\u2014Detailed Performance & Feature Comparison. n.d. https:\/\/docsbot.ai\/models\/compare\/gpt-4o\/deepseek-v3"},{"key":"e_1_3_4_34_2","volume-title":"The Total Cost of DeepSeek's AI Models Exceeded $1.5 Billion, Report Estimates","author":"Anonymous","year":"2025","unstructured":"Anonymous: The Total Cost of DeepSeek's AI Models Exceeded $1.5 Billion, Report Estimates. 2025. https:\/\/cybernews.com\/news\/cost-of-deepseeks-ai-models\/"},{"key":"e_1_3_4_35_2","volume-title":"How to Evaluate the RAG Pipeline Cost?","author":"Kumar V","year":"2024","unstructured":"Kumar V: How to Evaluate the RAG Pipeline Cost? 2024. https:\/\/adasci.org\/how-to-evaluate-the-rag-pipeline-cost\/"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","unstructured":"Bandyopadhyay D Bhattacharjee S Ekbal A: Thinking machines: A survey of LLM based reasoning strategies. 2025. 10.48550\/arXiv.2503.10814","DOI":"10.48550\/arXiv.2503.10814"},{"key":"e_1_3_4_37_2","first-page":"31210","volume-title":"Proceedings of the 40th International Conference on Machine Learning","author":"Shi F","year":"2023","unstructured":"Shi F, Chen X, Misra K, et al: Large language models can be easily distracted by irrelevant context, in Krause A, Brunskill E, Cho K, et al (eds): Proceedings of the 40th International Conference on Machine Learning, 2023. pp 31210-31227. Proceedings of Machine Learning Research PMLR"}],"container-title":["JCO Clinical Cancer Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ascopubs.org\/doi\/pdfdirect\/10.1200\/CCI-25-00220","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,14]],"date-time":"2025-11-14T21:00:30Z","timestamp":1763154030000},"score":1,"resource":{"primary":{"URL":"https:\/\/ascopubs.org\/doi\/10.1200\/CCI-25-00220"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11]]},"references-count":36,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["10.1200\/CCI-25-00220"],"URL":"https:\/\/doi.org\/10.1200\/cci-25-00220","relation":{},"ISSN":["2473-4276"],"issn-type":[{"value":"2473-4276","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11]]},"assertion":[{"value":"2025-07-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-02","order":1,"name":"revised","label":"Revised","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2500220"}}