{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T05:09:47Z","timestamp":1779167387278,"version":"3.51.4"},"reference-count":33,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T00:00:00Z","timestamp":1779148800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Oncol."],"abstract":"<jats:sec>\n                    <jats:title>Background\/objectives<\/jats:title>\n                    <jats:p>Large Language Models (LLMs) are increasingly used in medicine for tasks ranging from patient communication to exam preparation. This study aimed to evaluate the feasibility of using a domain-specific, out-of-training-data radiation and cancer biology examination as a benchmarking framework for large language models, and to compare the accuracy and consistency of commonly used LLMs available at the time of data collection.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>GPT-3.5, GPT-4, and Llama-2 were queried with 335 multiple-choice questions (MCQs) from the 2023 American Society for Radiation Oncology (ASTRO) Radiation and Cancer Biology Exam Study Guide, excluding image-based items. Each model answered all questions five times over three months to evaluate consistency. Model responses were scored against the official answer key and analyzed using one-way ANOVA with Bonferroni correction to determine statistical differences in accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>GPT-4 achieved the highest accuracy, correctly answering 81% of questions, significantly outperforming GPT-3.5 (62%) and Llama-2 (51%) (p &amp;lt; 0.001). All models performed worse on questions requiring calculations, though differences were not statistically significant. In terms of reliability, GPT-4 and Llama-2 provided consistent responses more frequently than GPT-3.5. Despite stable overall scores, all models exhibited variability in individual responses across repeated trials. GPT-4 produced the longest explanations, averaging 183 words per answer.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This study demonstrates the feasibility of using a domain-specific, out-of-training-data examination to benchmark large language model knowledge in radiation and cancer biology. While performance differences were observed among models, variability and limitations, particularly in calculation-based questions, highlight the importance of methodological benchmarking and cautious interpretation when considering medical educational applications.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fonc.2026.1738955","type":"journal-article","created":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T04:21:28Z","timestamp":1779164488000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Performance of large language models on the radiation and cancer biology practice exam"],"prefix":"10.3389","volume":"16","author":[{"given":"Jessica","family":"Bertschmann","sequence":"first","affiliation":[{"name":"Division of Radiation Oncology, Arthur Child Cancer Centre","place":["Calgary, AB, Canada"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"Xu","sequence":"additional","affiliation":[{"name":"Division of Radiation Oncology, Arthur Child Cancer Centre","place":["Calgary, AB, Canada"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Conrad","family":"Bayley","sequence":"additional","affiliation":[{"name":"Division of Radiation Oncology, Arthur Child Cancer Centre","place":["Calgary, AB, Canada"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ahmad","family":"Abdellatif","sequence":"additional","affiliation":[{"name":"Department of Electrical & Software Engineering, University of Calgary","place":["Calgary, AB, Canada"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sangjune Laurence","family":"Lee","sequence":"additional","affiliation":[{"name":"Division of Radiation Oncology, Arthur Child Cancer Centre","place":["Calgary, AB, Canada"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2026,5,19]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.65215\/ctdc8e75","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PloS Digit Health"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.46721","article-title":"Performance of large language models on a neurology board\u2013style examination","volume":"6","author":"Schubert","year":"2023","journal-title":"JAMA Netw Open"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.17641","article-title":"Performance of large language models on medical oncology examination questions","volume":"7","author":"Longwell","year":"2024","journal-title":"JAMA Netw Open"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1016\/j.anndiagpath.2024.152392","article-title":"Large language models in pathology: a comparative study of ChatGPT and bard with pathology trainees on multiple-choice questions","volume":"73","author":"Du","year":"2024","journal-title":"Ann Diagn Pathol"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1093\/ced\/llad355","article-title":"Performance of large language model artificial intelligence on dermatology board exam questions","volume":"49","author":"Park","year":"2024","journal-title":"Clin Exp Dermatol"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1016\/j.acra.2024.06.046","article-title":"Large language models as tools to generate radiology board-style multiple-choice questions","volume":"31","author":"Mistry","year":"2024","journal-title":"Acad Radiol"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2023.03.075","article-title":"ChatGPT: Can a natural language processing tool be trusted for radiation oncology use","volume":"116","author":"Ebrahimi","year":"2023","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.4630","article-title":"Quality of large language model responses to radiation oncology patient care questions","volume":"7","author":"Yalamanchili","year":"2024","journal-title":"JAMA Netw Open"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2023.06.2497","article-title":"Evaluating the reliability of Chat-GPT model responses for radiation oncology patient inquiries","volume":"117","author":"Floyd","year":"2023","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B12","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41523-023-00557-8","article-title":"Large language model (ChatGPT) as a support tool for breast tumor board","volume":"9","author":"Sorin","year":"2023","journal-title":"NPJ Breast Cancer"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2023.06.1727","article-title":"Evaluating the performance of ChatGPT at breast tumor board","volume":"117","author":"Xu","year":"2023","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.44541","article-title":"A radiation oncology board exam of ChatGPT","volume":"15","author":"Barbour","year":"2023","journal-title":"Cureus"},{"key":"B15","doi-asserted-by":"publisher","first-page":"101400","DOI":"10.1016\/j.adro.2023.101400","article-title":"Exploring capabilities of large language models such as ChatGPT in radiation oncology","volume":"9","author":"Dennst\u00e4dt","year":"2024","journal-title":"Adv Radiat Oncol"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacr.2024.07.011","article-title":"The accuracy of artificial intelligence ChatGPT in oncology examination questions","volume":"21","author":"Chow","year":"2024","journal-title":"J Am Coll Radiol"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1265024","article-title":"Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for AI-assisted medical education and decision making in radiation oncology","volume":"13","author":"Huang","year":"2023","journal-title":"Front Oncol"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.35924","article-title":"Evaluating the application of large language models in clinical research contexts","volume":"6","author":"Perlis","year":"2023","journal-title":"JAMA Netw Open"},{"key":"B19","volume-title":"ASTRO radiation and cancer biology study guide","author":"Huber","year":"2023"},{"key":"B20","unstructured":"Large language model LLaMA\n          \n          \n          \n          2023"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08774","article-title":"GPT-4 technical report","author":"Achiam","year":"2023","journal-title":"arXiv Preprint arXiv:2303.08774"},{"key":"B23","doi-asserted-by":"publisher","first-page":"2050","DOI":"10.1038\/s41467-024-46411-8","article-title":"Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks","volume":"15","author":"Sandmann","year":"2024","journal-title":"Nat Commun"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1219326","article-title":"Evaluating large language models on a highly-specialized topic, radiation oncology physics","volume":"13","author":"Holmes","year":"2023","journal-title":"Front Oncol"},{"key":"B25","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1097\/js9.0000000000001850","article-title":"The performance of large language model powered chatbots compared to oncology physicians on colorectal cancer queries","author":"Zhou","year":"2024","journal-title":"Int J Surg"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.05398","article-title":"Mathprompter: Mathematical reasoning using large language models","author":"Imani","year":"2023","journal-title":"arXiv Preprint arXiv:2303.05398"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.2308.02828","article-title":"LLM is like a box of chocolates: the non-determinism of ChatGPT in code generation","author":"Ouyang","year":"2023","journal-title":"arXiv Preprint arXiv:2308.02828"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1016\/j.surg.2023.12.014","article-title":"Evaluating capabilities of large language models: performance of GPT-4 on surgical knowledge assessments","volume":"175","author":"Beaulieu-Jones","year":"2024","journal-title":"Surgery"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3662681","article-title":"One vs. many: comprehending accurate information from multiple erroneous and inconsistent AI generations","author":"Lee","year":"2024","journal-title":"Proc ACM Conf Fairness Account Transpar"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-07421-0","article-title":"Detecting hallucinations in large language models using semantic entropy","volume":"630","author":"Farquhar","year":"2024","journal-title":"Nature"},{"key":"B31","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarization","volume":"8","author":"Asgari","year":"2025","journal-title":"NPJ Digital Med"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf008","article-title":"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines","volume":"32","author":"Liu","year":"2025","journal-title":"J Am Med Inf Assoc"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat Med"}],"container-title":["Frontiers in Oncology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2026.1738955\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T04:21:31Z","timestamp":1779164491000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2026.1738955\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,19]]},"references-count":33,"alternative-id":["10.3389\/fonc.2026.1738955"],"URL":"https:\/\/doi.org\/10.3389\/fonc.2026.1738955","relation":{},"ISSN":["2234-943X"],"issn-type":[{"value":"2234-943X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,19]]},"article-number":"1738955"}}