{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T09:40:22Z","timestamp":1783676422543,"version":"3.55.0"},"reference-count":17,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T00:00:00Z","timestamp":1768003200000},"content-version":"vor","delay-in-days":9,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,1,28]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models (LLMs), such as ChatGPT and Grok, have rapidly advanced in natural language understanding and are increasingly being applied to specialized fields, including medicine. In this study, we evaluated the domain-specific knowledge of LLMs in radiotherapy by assessing their performance on three certification examinations in Japan: the Japanese Medical Physicist Examination, the Japanese Board Examination for Radiologists and the Japanese Board Examination for Radiation Oncologists. We assessed five LLMs\u2014ChatGPT-5, ChatGPT-5 Pro, Grok 4, Grok 4 heavy and Gemini 2.5 Pro\u2014by inputting all multiple-choice questions from these exams into each model and recording their responses. The AI-generated answers were compared with reference answers determined by experienced medical physicists and radiation oncologists. The results demonstrated average accuracies of 84.7\u2009\u00b1\u20092.0% (ChatGPT-5), 94.7\u2009\u00b1\u20092.1% (ChatGPT-5 Pro), 78.4\u2009\u00b1\u20091.2% (Grok 4), 81.6\u2009\u00b1\u20092.2% (Grok 4 heavy) and 88.9\u2009\u00b1\u20091.2% (Gemini 2.5 Pro). All models achieved over 75% accuracy, with ChatGPT-5 Pro consistently outperforming others, attaining an average accuracy exceeding 90% across all examinations. These findings highlight the strong potential of advanced LLMs, particularly ChatGPT-5 Pro, for future integration into radiotherapy-related applications such as automated contouring and treatment planning support.<\/jats:p>","DOI":"10.1093\/jrr\/rraf083","type":"journal-article","created":{"date-parts":[[2025,11,19]],"date-time":"2025-11-19T13:17:01Z","timestamp":1763558221000},"page":"114-120","source":"Crossref","is-referenced-by-count":3,"title":["Evaluating the capability of large language models in radiotherapy through professional certification examinations in Japan"],"prefix":"10.1093","volume":"67","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5018-3800","authenticated-orcid":false,"given":"Noriyuki","family":"Kadoya","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoshiyuki","family":"Takahashi","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seiya","family":"Koga","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hikaru","family":"Tanno","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kazuhiro","family":"Arai","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shohei","family":"Tanaka","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5984-9226","authenticated-orcid":false,"given":"Yoshiyuki","family":"Katsuta","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hinako","family":"Harada","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"So","family":"Omata","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3562-1037","authenticated-orcid":false,"given":"Takaya","family":"Yamamoto","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rei","family":"Umezawa","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2906-5575","authenticated-orcid":false,"given":"Ken","family":"Takeda","sequence":"additional","affiliation":[{"name":"Course of Radiological Technology, Health Sciences, Tohoku University Graduate School of Medicine , 2-1 Seiryo-machi, Aoba-ku, Sendai, 980-8575,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7032-1577","authenticated-orcid":false,"given":"Keiichi","family":"Jingu","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Tohoku University Graduate School of Medicine , 1-1 Seiryo-machi, Aoba-ku, Sendai, Miyagi 980-8574 ,","place":["Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,1,10]]},"reference":[{"key":"2026013002565810400_ref1","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2026013002565810400_ref2","article-title":"Traceable evidence enhanced visual grounded reasoning: evaluation and methodology","author":"Wang","year":"2025"},{"key":"2026013002565810400_ref3","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics15111320","article-title":"Do LLMs have \u2018the eye\u2019 for MRI? Evaluating GPT-4o, Grok, and Gemini on brain MRI performance: first evaluation of Grok in medical imaging and a comparative analysis","volume":"15","author":"Sozer","year":"2025","journal-title":"Diagnostics (Basel)"},{"key":"2026013002565810400_ref4","article-title":"Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities","author":"Comanici","year":"2025"},{"key":"2026013002565810400_ref5","doi-asserted-by":"publisher","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"key":"2026013002565810400_ref6","article-title":"Towards expert-level medical question answering with large language models","author":"Singhal","year":"2023"},{"key":"2026013002565810400_ref7","doi-asserted-by":"publisher","first-page":"e39443","DOI":"10.2196\/39443","article-title":"Learning the treatment process in radiotherapy using an artificial intelligence-assisted chatbot: development study","volume":"6","author":"Rebelo","year":"2022","journal-title":"JMIR Form Res"},{"key":"2026013002565810400_ref8","article-title":"Radiology-GPT: a large language model for radiology","author":"Liu","year":"2023"},{"key":"2026013002565810400_ref9","doi-asserted-by":"publisher","first-page":"101400","DOI":"10.1016\/j.adro.2023.101400","article-title":"Exploring capabilities of large language models such as ChatGPT in radiation oncology","volume":"9","author":"Dennst\u00e4dt","year":"2024","journal-title":"Adv Radiat Oncol"},{"key":"2026013002565810400_ref10","doi-asserted-by":"publisher","first-page":"1557064","DOI":"10.3389\/fonc.2025.1557064","article-title":"A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options","volume":"15","author":"Wang","year":"2025","journal-title":"Front Oncol"},{"key":"2026013002565810400_ref11","doi-asserted-by":"publisher","first-page":"929","DOI":"10.1007\/s12194-024-00838-2","article-title":"Assessing knowledge about medical physics in language-generative AI with large language model: using the medical physicist exam","volume":"17","author":"Kadoya","year":"2024","journal-title":"Radiol Phys Technol"},{"key":"2026013002565810400_ref12","doi-asserted-by":"publisher","first-page":"075007","DOI":"10.1088\/1361-6560\/adbff1","article-title":"A feasibility study of automating radiotherapy planning with large language model agents","volume":"70","author":"Wang","year":"2025","journal-title":"Phys Med Biol"},{"key":"2026013002565810400_ref13","doi-asserted-by":"publisher","first-page":"9186","DOI":"10.1038\/s41467-024-53387-y","article-title":"LLM-driven multimodal target volume contouring in radiation oncology","volume":"15","author":"Oh","year":"2024","journal-title":"Nat Commun"},{"key":"2026013002565810400_ref14","doi-asserted-by":"publisher","first-page":"619","DOI":"10.1002\/mp.17416","article-title":"Large-language-model empowered 3D dose prediction for intensity-modulated radiotherapy","volume":"52","author":"Dong","year":"2025","journal-title":"Med Phys"},{"key":"2026013002565810400_ref15","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","author":"Liu","year":"2021"},{"key":"2026013002565810400_ref16","doi-asserted-by":"publisher","first-page":"706","DOI":"10.1007\/s11604-024-01705-1","article-title":"Application of notebook LM, a large language model with retrieval-augmented generation, for lung cancer staging","volume":"43","author":"Tozuka","year":"2025","journal-title":"Jpn J Radiol"},{"key":"2026013002565810400_ref17","article-title":"Asking again and again: exploring LLM robustness to repeated questions","author":"Shaier","year":"2024"}],"container-title":["Journal of Radiation Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jrr\/article-pdf\/67\/1\/114\/66339684\/rraf083.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrr\/article-pdf\/67\/1\/114\/66339684\/rraf083.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T07:57:06Z","timestamp":1769759826000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrr\/article\/67\/1\/114\/8419675"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1]]},"references-count":17,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1,10]]},"published-print":{"date-parts":[[2026,1,28]]}},"URL":"https:\/\/doi.org\/10.1093\/jrr\/rraf083","relation":{},"ISSN":["0449-3060","1349-9157"],"issn-type":[{"value":"0449-3060","type":"print"},{"value":"1349-9157","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,1]]},"published":{"date-parts":[[2026,1]]}}}