{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T12:31:20Z","timestamp":1792758680029,"version":"4.5.0"},"reference-count":53,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T00:00:00Z","timestamp":1694649600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Oncol."],"abstract":"<jats:sec><jats:title>Purpose<\/jats:title><jats:p>The potential of large language models in medicine for education and decision-making purposes has been demonstrated as they have achieved decent scores on medical exams such as the United States Medical Licensing Exam (USMLE) and the MedQA exam. This work aims to evaluate the performance of ChatGPT-4 in the specialized field of radiation oncology.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>The 38th American College of Radiology (ACR) radiation oncology in-training (TXIT) exam and the 2022 Red Journal Gray Zone cases are used to benchmark the performance of ChatGPT-4. The TXIT exam contains 300 questions covering various topics of radiation oncology. The 2022 Gray Zone collection contains 15 complex clinical cases.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>For the TXIT exam, ChatGPT-3.5 and ChatGPT-4 have achieved the scores of 62.05% and 78.77%, respectively, highlighting the advantage of the latest ChatGPT-4 model. Based on the TXIT exam, ChatGPT-4\u2019s strong and weak areas in radiation oncology are identified to some extent. Specifically, ChatGPT-4 demonstrates better knowledge of statistics, CNS &amp;amp; eye, pediatrics, biology, and physics than knowledge of bone &amp;amp; soft tissue and gynecology, as per the ACR knowledge domain. Regarding clinical care paths, ChatGPT-4 performs better in diagnosis, prognosis, and toxicity than brachytherapy and dosimetry. It lacks proficiency in in-depth details of clinical trials. For the Gray Zone cases, ChatGPT-4 is able to suggest a personalized treatment approach to each case with high correctness and comprehensiveness. Importantly, it provides novel treatment aspects for many cases, which are not suggested by any human experts.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Both evaluations demonstrate the potential of ChatGPT-4 in medical education for the general public and cancer patients, as well as the potential to aid clinical decision-making, while acknowledging its limitations in certain domains. Owing to the risk of hallucinations, it is essential to verify the content generated by models such as ChatGPT for accuracy.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fonc.2023.1265024","type":"journal-article","created":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T18:51:36Z","timestamp":1694717496000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":114,"title":["Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology"],"prefix":"10.3389","volume":"13","author":[{"given":"Yixing","family":"Huang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Gomaa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sabine","family":"Semrau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marlen","family":"Haderlein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sebastian","family":"Lettmaier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Weissmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johanna","family":"Grigo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hassen Ben","family":"Tkhayat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Frey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Udo","family":"Gaipl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luitpold","family":"Distel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Maier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rainer","family":"Fietkau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph","family":"Bert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Florian","family":"Putz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,9,14]]},"reference":[{"key":"B1","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"B2","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"B3","article-title":"Chain of thought prompting elicits reasoning in large language models","author":"Wei","year":"2022","journal-title":"NeurIPS"},{"key":"B4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10439-023-03284-0#citeas","article-title":"ChatGPT, bard, and large language models for biomedical research: opportunities and pitfalls","author":"Thapa","year":"2023","journal-title":"Ann Biomed Eng"},{"key":"B5","article-title":"LLaMA: Open and efficient foundation language models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2302.13971"},{"key":"B6","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nat"},{"key":"B7","article-title":"Chat with ChatGPT on industry 5.0: Learning and decision-making for intelligent industries","volume":"10","author":"Wang","year":"2023","journal-title":"IEEE\/CAA J Autom Sin"},{"key":"B8","doi-asserted-by":"crossref","DOI":"10.1016\/j.ijrobp.2023.03.075","article-title":"ChatGPT: Can a natural language processing tool be trusted for radiation oncology use","volume":"116","author":"Ebrahimi","year":"2023","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B9","first-page":"1","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM Trans Comput Healthc (Health)"},{"key":"B10","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.40895","article-title":"Chatdoctor: A medical chat model fine-tuned on a large language model Meta-AI (LLaMA) using medical domain knowledge","volume":"15","author":"Li","year":"2023","journal-title":"Cureus"},{"key":"B12","article-title":"Publicly available clinical BERT embeddings","author":"Alsentzer","year":"2019","journal-title":"Proc Clin NLP Workshop"},{"key":"B13","article-title":"Huatuo: Tuning LLaMA model with chinese medical knowledge","author":"Wang","year":"2023","journal-title":"arXiv preprint arXiv:2304.06975"},{"key":"B14","article-title":"ChatCAD: Interactive computer-aided diagnosis on medical image using large language models","author":"Wang","year":"2023","journal-title":"arXiv preprint arXiv:2302.07257"},{"key":"B15","article-title":"GPT-4 technical report","year":"2023","journal-title":"arXiv preprint arXiv:2303.08774"},{"key":"B16","article-title":"Deep reinforcement learning from human preferences","volume":"30","author":"Christiano","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"B17","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PloS Digit Health"},{"key":"B18","doi-asserted-by":"crossref","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"B19","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.1002\/mp.13264","article-title":"Deep learning in medical imaging and radiation therapy","volume":"46","author":"Sahiner","year":"2019","journal-title":"Med Phys"},{"key":"B20","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","article-title":"Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation","volume":"36","author":"Kamnitsas","year":"2017","journal-title":"Med Image Anal"},{"key":"B21","doi-asserted-by":"crossref","DOI":"10.1002\/mp.15863","article-title":"Deep learning for brain metastasis detection and segmentation in longitudinal mri data","volume":"49","author":"Huang","year":"2022","journal-title":"Med Phys"},{"key":"B22","doi-asserted-by":"crossref","DOI":"10.3389\/fonc.2023.1115258","article-title":"Deep learning for automatic head and neck lymph node level delineation provides expert-level accuracy","volume":"13","author":"Weissmann","year":"2023","journal-title":"Front Oncol"},{"key":"B23","first-page":"1","article-title":"Improving cbct image quality to the ct level using reggan in esophageal cancer adaptive radiotherapy","author":"Wang","year":"2023","journal-title":"Strahlenther Onkol"},{"key":"B24","doi-asserted-by":"crossref","DOI":"10.1002\/mp.13953","article-title":"A feasibility study on deep learning-based radiotherapy dose calculation","volume":"47","author":"Xing","year":"2020","journal-title":"Med Phys"},{"key":"B25","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/s00066-021-01874-2","article-title":"Computed tomography-based deep-learning prediction of induction chemotherapy treatment response in locally advanced nasopharyngeal carcinoma","volume":"198","author":"Yang","year":"2022","journal-title":"Strahlenther Onkol"},{"key":"B26","article-title":"Deep learning for cancer prognosis prediction using portrait photos by StyleGAN embedding","author":"Hagag","year":"2023","journal-title":"arXiv preprint arXiv:2306.14596"},{"key":"B27","doi-asserted-by":"crossref","DOI":"10.3389\/fonc.2023.1219326","article-title":"Evaluating large language models on a highly-specialized topic, radiation oncology physics","volume":"13","author":"Holmes","year":"2023","journal-title":"Front Oncol"},{"key":"B28","article-title":"Analysis of the radiation oncology in-training examination content using a clinical care path conceptual framework","volume":"10","author":"Rogacki","year":"2021","journal-title":"Appl Radiat Oncol"},{"key":"B29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijrobp.2016.11.052","article-title":"Introducing: The red journal gray zone","volume":"97","author":"Palma","year":"2017","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B30","first-page":"6","article-title":"GPT-4 system card","year":"2023","journal-title":"OpenAI.com"},{"key":"B31","doi-asserted-by":"crossref","DOI":"10.1200\/JCO.2019.37.15_suppl.TPS600","article-title":"NRG Oncology\/NSABP B-51\/RTOG 1304: Phase III trial to determine if chest wall and regional nodal radiotherapy (CWRNRT) post mastectomy (Mx) or the addition of RNRT to whole breast rt post breast-conserving surgery (BCS) reduces invasive breast cancer recurrence-free interval (IBCR-FI) in patients (pts) with pathologically positive axillary (PPAx) nodes who are ypN0 after neoadjuvant chemotherapy (NC)","volume":"37","author":"Mamounas","year":"2019","journal-title":"J Clin Oncol"},{"key":"B32","doi-asserted-by":"crossref","DOI":"10.1016\/S1470-2045(17)30086-4","article-title":"Optimal fractionation of preoperative radiotherapy and timing to surgery for rectal cancer (stockholm iii): a multicentre, randomised, non-blinded, phase 3, non-inferiority trial","volume":"18","author":"Erlandsson","year":"2017","journal-title":"Lancet Oncol"},{"key":"B33","doi-asserted-by":"crossref","DOI":"10.1016\/S1470-2045(18)30132-3","article-title":"Chemotherapy versus chemoradiotherapy after surgery and preoperative chemotherapy for resectable gastric cancer (critics): an international, open-label, randomised phase 3 trial","volume":"19","author":"Cats","year":"2018","journal-title":"Lancet Oncol"},{"key":"B34","doi-asserted-by":"crossref","DOI":"10.1016\/S1470-2045(19)30395-X","article-title":"Adjuvant chemoradiotherapy versus radiotherapy alone in women with high-risk endometrial cancer (portec-3): patterns of recurrence and post-hoc survival analysis of a randomised phase 3 trial","volume":"20","author":"de Boer","year":"2019","journal-title":"Lancet Oncol"},{"key":"B35","doi-asserted-by":"crossref","DOI":"10.1016\/j.radonc.2015.12.025","article-title":"Dose\u2013effect relationship and risk factors for vaginal stenosis after definitive radio (chemo) therapy with imageguided brachytherapy for locally advanced cervical cancer in the embrace study","volume":"118","author":"Kirchheiner","year":"2016","journal-title":"Radiothe Oncol"},{"key":"B36","doi-asserted-by":"crossref","DOI":"10.1001\/jamaoncol.2020.0147","article-title":"Outcomes of observation vs stereotactic ablative radiation for oligometastatic prostate cancer: the oriole phase 2 randomized clinical trial","volume":"6","author":"Phillips","year":"2020","journal-title":"JAMA Oncol"},{"key":"B37","article-title":"The eighth edition AJCC cancer staging manual: continuing to build a bridge from a population-based to a more \u201cpersonalized\u201d approach to cancer staging","volume":"67","author":"Amin","year":"2017","journal-title":"CA: Cancer J Clin"},{"key":"B38","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1016\/j.ijrobp.2020.09.036","article-title":"Cao J. A viewpoint on isolated contralateral axillary lymph node involvement by breast cancer: regional recurrence or distant metastasis","volume":"113","author":"Al-Rashdan","year":"2022","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B39","article-title":"Endocrine therapy for early breast cancer: updated review","volume":"17","author":"Tremont","year":"2017","journal-title":"Ochsner J"},{"key":"B40","doi-asserted-by":"crossref","DOI":"10.1016\/j.ijrobp.2022.08.033","article-title":"Sowing the seeds: A case of oligometastatic anal cancer 12 years after prostate brachytherapy","volume":"114","author":"Tchelebi","year":"2022","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B41","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.ijrobp.2021.03.013","article-title":"Exploring the role of resection post-radiation therapy in gliomas","volume":"113","author":"Scarpelli","year":"2022","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B42","doi-asserted-by":"crossref","DOI":"10.1016\/j.ijrobp.2019.05.028","article-title":"Radiation therapy for cure or palliation: Case of the immunosuppressed patient with multiple primary cancers and liver transplant","volume":"112","author":"Prpic","year":"2022","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B43","first-page":"4","article-title":"Synopsis of supraclavicular sarcoma: synthesis of stratagem and solutions","volume":"112","author":"Johnson","year":"2021","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B44","doi-asserted-by":"crossref","DOI":"10.1016\/j.ijrobp.2021.03.014","article-title":"Postoperative radiation therapy in prostate cancer: Timing, duration of hormonal treatment and the use of PSMA PET-CT","volume":"113","author":"Berghen","year":"2022","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B45","doi-asserted-by":"crossref","DOI":"10.1093\/neuonc\/now182","article-title":"Tumor treating fields: a novel treatment modality and its use in brain tumors","volume":"18","author":"Hottinger","year":"2016","journal-title":"Neuro-Oncol"},{"key":"B46","doi-asserted-by":"crossref","DOI":"10.1016\/j.ijrobp.2022.02.029","article-title":"One way or another: An oligorecurrence after an oligometastasis of an estrogen receptor-positive breast cancer","volume":"114","author":"Goodman","year":"2022","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"B47","doi-asserted-by":"crossref","DOI":"10.1007\/s00066-022-01939-w","article-title":"Digital transfer in radiation oncology education for medical students\u2014single-center data and systemic review of the literature","volume":"198","author":"Oertel","year":"2022","journal-title":"Strahlenther Onkol"},{"key":"B48","doi-asserted-by":"crossref","first-page":"e2214840120","DOI":"10.1073\/pnas.2214840120","article-title":"Superhuman artificial intelligence can improve human decision-making by increasing novelty","volume":"120","author":"Shin","year":"2023","journal-title":"PNAS"},{"key":"B49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13054-023-04393-x","article-title":"Large language models and the perils of their hallucinations","volume":"27","author":"Azamfirei","year":"2023","journal-title":"Crit Care"},{"key":"B50","first-page":"1","article-title":"GPT-4: a new era of artificial intelligence in medicine","author":"Waisberg","year":"2023","journal-title":"Ir J Med Sci"},{"key":"B51","first-page":"1","article-title":"Large language model ai chatbots require approval as medical devices","author":"Gilbert","year":"2023","journal-title":"Nat Med"},{"key":"B52","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2307.09288"},{"key":"B53","article-title":"Towards expert-level medical question answering with large language models","author":"Singhal","year":"2023","journal-title":"arXiv preprint arXiv:2305.09617"}],"container-title":["Frontiers in Oncology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2023.1265024\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,14]],"date-time":"2023-09-14T18:51:56Z","timestamp":1694717516000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2023.1265024\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,14]]},"references-count":53,"alternative-id":["10.3389\/fonc.2023.1265024"],"URL":"https:\/\/doi.org\/10.3389\/fonc.2023.1265024","relation":{},"ISSN":["2234-943X"],"issn-type":[{"value":"2234-943X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,14]]},"article-number":"1265024"}}