{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,29]],"date-time":"2026-08-29T10:18:56Z","timestamp":1787998736372,"version":"build-2784847793"},"reference-count":42,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T00:00:00Z","timestamp":1749513600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Cancer Institute","award":["75N91023C00051"],"award-info":[{"award-number":["75N91023C00051"]}]},{"name":"National Cancer Institute","award":["90966179"],"award-info":[{"award-number":["90966179"]}]},{"DOI":"10.13039\/100000871","name":"Mayo Clinic Marley Endowment Fund","doi-asserted-by":"publisher","award":["75N91023C00051"],"award-info":[{"award-number":["75N91023C00051"]}],"id":[{"id":"10.13039\/100000871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000871","name":"Mayo Clinic Marley Endowment Fund","doi-asserted-by":"publisher","award":["90966179"],"award-info":[{"award-number":["90966179"]}],"id":[{"id":"10.13039\/100000871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Cancers"],"abstract":"<jats:p>Background\/Objectives: The accurate delineation of primary tumors (GTVp) and metastatic lymph nodes (GTVn) in head and neck (HN) cancers is essential for effective radiation treatment planning, yet remains a challenging and laborious task. This study aims to develop a deep-learning-based auto-segmentation (DLAS) model trained on external datasets with false-positive elimination using clinical diagnosis reports. Methods: The DLAS model was trained on a multi-institutional public dataset with 882 cases. Forty-four institutional cases were randomly selected as the external testing dataset. DLAS-generated GTVp and GTVn were validated against clinical diagnosis reports to identify false-positive and false-negative segmentation errors using two large language models: ChatGPT-4 and Llama-3. False-positive ruling out was conducted by matching the centroids of AI-generated contours with the slice locations or anatomical regions described in the reports. Performance was evaluated using the Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), and tumor detection precision. Results: ChatGPT-4 outperformed Llama-3 in accurately extracting tumor locations from the diagnostic reports. False-positive contours were identified in 15 out of 44 cases. The DSCmean of the DLAS contours for GTVp and GTVn increased from 0.68 to 0.75 and from 0.69 to 0.75, respectively, after the ruling-out process. Notably, the average HD95 value for GTVn decreased from 18.81 mm to 5.2 mm. Post ruling out, the model achieved 100% precision for GTVp and GTVn when compared with the results of physician-determined contours. Conclusions: The false-positive ruling-out approach based on diagnostic reports effectively enhances the precision of DLAS in the HN region. The model accurately identifies the tumor location and detects all false-negative errors.<\/jats:p>","DOI":"10.3390\/cancers17121935","type":"journal-article","created":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T12:53:16Z","timestamp":1749559996000},"page":"1935","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Improving the Precision of Deep-Learning-Based Head and Neck Target Auto-Segmentation by Leveraging Radiology Reports Using a Large Language Model"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9780-0095","authenticated-orcid":false,"given":"Libing","family":"Zhu","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ 85058, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Claude M.","family":"Rwigema","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ 85058, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2181-9889","authenticated-orcid":false,"given":"Xue","family":"Feng","sequence":"additional","affiliation":[{"name":"Carina Medical LLC., Lexington, KY 40513, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bilaal","family":"Ansari","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ 85058, USA"},{"name":"Department of Physics, Lake Forest College, Lake Forest, IL 60045, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6551-9127","authenticated-orcid":false,"given":"Jingwei","family":"Duan","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, The University of Alabama at Birmingham, Birmingham, AL 35233, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2620-1893","authenticated-orcid":false,"given":"Yi","family":"Rong","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ 85058, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Quan","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Mayo Clinic, Phoenix, AZ 85058, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1001\/jamaoncol.2016.5688","article-title":"Global, Regional, and National Cancer Incidence, Mortality, Years of Life Lost, Years Lived with Disability, and Disability-Adjusted Life-years for 32 Cancer Groups, 1990 to 2015, A Systematic Analysis for the Global Burden of Disease Study","volume":"3","author":"Javanbakht","year":"2017","journal-title":"JAMA Oncol."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Andrearczyk, V., Oreiller, V., Boughdad, S., Rest, C.C.L., Elhalawani, H., Jreige, M., Prior, J.O., Valli\u00e8res, M., Visvikis, D., and Hatt, M. (2022). Overview of the HECKTOR Challenge at MICCAI 2021, Automatic Head and Neck Tumor Segmentation and Outcome Prediction in PET\/CT Images, Springer International Publishing.","DOI":"10.1007\/978-3-030-98253-9_1"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1016\/j.meddos.2022.09.004","article-title":"Clinical target volume segmentation based on gross tumor volume using deep learning for head and neck cancer treatment","volume":"48","author":"Kihara","year":"2023","journal-title":"Med. Dosim."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.radonc.2019.04.006","article-title":"Interobserver variability in delineation of target volumes in head and neck cancer","volume":"137","author":"Gulyban","year":"2019","journal-title":"Radiother. Oncol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.jmir.2016.11.003","article-title":"Interobserver variability in the delineation of gross tumour volume and specified organs-at-risk during IMRT for head and neck cancers and the impact of FDG-PET\/CT on such variability at the primary site","volume":"48","author":"Gudi","year":"2017","journal-title":"J. Med. Imaging Radiat. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2996","DOI":"10.1200\/JCO.2009.27.4498","article-title":"Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: Results from TROG 02.02","volume":"28","author":"Peters","year":"2010","journal-title":"J. Clin. Oncol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1016\/j.ijrobp.2010.10.019","article-title":"Variations in the contouring of organs at risk: Test case from a patient with oropharyngeal cancer","volume":"82","author":"Nelms","year":"2012","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1060","DOI":"10.1016\/j.ijrobp.2010.10.035","article-title":"Validating the RTOG-Endorsed Brachial Plexus Contouring Atlas: An Evaluation of Reproducibility Among Patients Treated by Intensity-Modulated Radiotherapy for Head-and-Neck Cancer","volume":"82","author":"Sun","year":"2012","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Andrearczyk, V., Oreiller, V., Hatt, M., and Depeursinge, A. (2022). Head and Neck Tumor Segmentation and Outcome Prediction, Springer.","DOI":"10.1007\/978-3-030-98253-9"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"110588","DOI":"10.1016\/j.radonc.2024.110588","article-title":"Achieving accurate prostate auto-segmentation on CT in the absence of MR imaging","volume":"202","author":"Duan","year":"2025","journal-title":"Radiother. Oncol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"6673","DOI":"10.1002\/mp.16768","article-title":"Contour subregion error detection methodology using deep learning auto-segmentation","volume":"50","author":"Duan","year":"2023","journal-title":"Med. Phys."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"109870","DOI":"10.1016\/j.radonc.2023.109870","article-title":"Evaluation of different algorithms for automatic segmentation of head-and-neck lymph nodes on CT images","volume":"188","author":"Costea","year":"2023","journal-title":"Radiother. Oncol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4568","DOI":"10.1002\/mp.13141","article-title":"Autosegmentation for thoracic radiation treatment planning: A grand challenge at AAPM 2017","volume":"45","author":"Yang","year":"2018","journal-title":"Med. Phys."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Myronenko, A., Siddiquee, M.M.R., Yang, D., He, Y., and Xu, D. (2023). Automated Head and Neck Tumor Segmentation from 3D PET\/CT HECKTOR 2022 Challenge Report. Head and Neck Tumor Segmentation and Outcome Prediction, Springer Nature.","DOI":"10.1007\/978-3-031-27420-6_2"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kaczmarska, M., and Majek, K. (2023, January 8). 3D Segmentation of Kidneys, Kidney Tumors and Cysts on CT Images\u2014KiTS23 Challenge. Proceedings of the Kidney and Kidney Tumor Segmentation: MICCAI 2023 Challenge, KiTS 2023, Vancouver, BC, Canada.","DOI":"10.1007\/978-3-031-54806-2_21"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Andrearczyk, V., Oreiller, V., Jreige, M., Vallieres, M., Castelli, J., Elhalawani, H., Boughdad, S., Prior, J.O., and Depeursinge, A. (2020, January 4). Overview of the HECKTOR challenge at MICCAI 2020, automatic head and neck tumor segmentation in PET\/CT. Proceedings of the Head and Neck Tumor Segmentation: First Challenge, HECKTOR 2020, Lima, Peru. Proceedings 1.","DOI":"10.1007\/978-3-030-67194-5_1"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1143","DOI":"10.1016\/j.jacr.2021.03.005","article-title":"Diagnostic accuracy and failure mode analysis of a deep learning algorithm for the detection of intracranial hemorrhage","volume":"18","author":"Voter","year":"2021","journal-title":"J. Am. Coll. Radiol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1016\/j.clon.2023.01.016","article-title":"A Review of the Metrics Used to Assess Auto-Contouring Systems in Radiotherapy","volume":"35","author":"Mackay","year":"2023","journal-title":"Clin. Oncol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1053\/j.semnuclmed.2021.06.008","article-title":"Limitations and Pitfalls of FDG-PET\/CT in Infection and Inflammation","volume":"51","author":"Jordy","year":"2021","journal-title":"Semin. Nucl. Med."},{"key":"ref_21","first-page":"1406","article-title":"Contextual Radiology Reporting: A New Approach to Neuroradiology Structured Templates","volume":"39","author":"Mamlouk","year":"2018","journal-title":"AJNR Am. J. Neuroradiol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e232714","DOI":"10.1148\/radiol.232714","article-title":"Potential of GPT-4 for Detecting Errors in Radiology Reports: Implications for Reporting Accuracy","volume":"311","author":"Gertz","year":"2024","journal-title":"Radiology"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.clinimag.2023.06.008","article-title":"Decoding radiology reports: Potential application of OpenAI ChatGPT to enhance patient understanding of diagnostic reports","volume":"101","author":"Hanzhou","year":"2023","journal-title":"Clin. Imaging"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1938","DOI":"10.1007\/s00330-024-11032-8","article-title":"Comparative analysis of GPT-4-based ChatGPT\u2019s diagnostic performance with radiologists using real-world radiology reports of brain tumors","volume":"35","author":"Mitsuyama","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kim, S.H., Schramm, S., Adams, L.C., Braren, R., Bressem, K.K., Keicher, M., Zimmer, C., Hedderich, D.M., and Wiestler, B. (2024). Performance of Open-Source LLMs in Challenging Radiological Cases\u2014A Benchmark Study on 1933 Eurorad Case Reports. medRxiv.","DOI":"10.1101\/2024.09.04.24313026"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhu, L., Rong, Y., McGee, L.A., Rwigema, J.M., and Patel, S.H. (2024). Testing and Validation of a Custom Retrained Large Language Model for the Supportive Care of HN Patients with External Knowledge Base. Cancers, 16.","DOI":"10.3390\/cancers16132311"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1016\/j.ijrobp.2023.10.033","article-title":"NRG Oncology Assessment of Artificial Intelligence Deep Learning\u2013Based Auto-segmentation for Radiation Therapy: Current Developments, Clinical Considerations, and Future Directions","volume":"119","author":"Rong","year":"2024","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","article-title":"nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation","volume":"18","author":"Isensee","year":"2021","journal-title":"Nat. Methods"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhao, L.-M., Zhang, H., Kim, D.D., Ghimire, K., Hu, R., Kargilis, D.C., Tang, L., Meng, S., Chen, Q., and Liao, W.-H. (2023). Head and neck tumor segmentation convolutional neural network robust to missing PET\/CT modalities using channel dropout. Phys. Med. Biol., 68.","DOI":"10.1088\/1361-6560\/accac9"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ghimire, K., Chen, Q., and Feng, X. (2022). Head and Neck Tumor Segmentation with Deeply-Supervised 3D UNet and Progression-Free Survival Prediction with Linear Model. Head and Neck Tumor Segmentation and Outcome Prediction, Springer International Publishing.","DOI":"10.1007\/978-3-030-98253-9_13"},{"key":"ref_31","unstructured":"Gai, Z., Tong, M., and Li, J. (2024, October 30). Chuanhu Chat (Software). Available online: https:\/\/github.com\/GaiZhenbiao\/ChuanhuChatGPT."},{"key":"ref_32","unstructured":"Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. (2024). The llama 3 herd of models. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"766","DOI":"10.1016\/j.ijrobp.2015.12.002","article-title":"Patterns of Recurrence in Electively Irradiated Lymph Node Regions After Definitive Accelerated Intensity Modulated Radiation Therapy for Head and Neck Squamous Cell Carcinoma","volume":"94","author":"Dijkema","year":"2016","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"ref_34","first-page":"1","article-title":"A systematic review on imbalanced data challenges in machine learning: Applications and solutions","volume":"52","author":"Kaur","year":"2019","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1007730.1007733","article-title":"Special issue on learning from imbalanced data sets","volume":"6","author":"Chawla","year":"2004","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_36","first-page":"25","article-title":"Handling imbalanced datasets: A review","volume":"30","author":"Kotsiantis","year":"2006","journal-title":"GESTS Int. Trans. Comput. Sci. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"4280","DOI":"10.1118\/1.2361076","article-title":"Tumor delineation using PET in head and neck cancers: Threshold contouring and lesion volumes","volume":"33","author":"Ford","year":"2006","journal-title":"Med. Phys."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2165","DOI":"10.1007\/s00259-010-1423-3","article-title":"PET-guided delineation of radiation therapy treatment volumes: A survey of image segmentation techniques","volume":"37","author":"Zaidi","year":"2010","journal-title":"Eur. J. Nucl. Med. Mol. Imaging"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3711","DOI":"10.1118\/1.2956713","article-title":"A novel PET tumor delineation method based on adaptive region-growing and dual-front active contours","volume":"35","author":"Li","year":"2008","journal-title":"Med. Phys."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1109\/83.557353","article-title":"Estimation of fuzzy Gaussian mixture and unsupervised statistical image segmentation","volume":"6","author":"Caillol","year":"1997","journal-title":"IEEE Trans. Image Process."},{"key":"ref_41","first-page":"40","article-title":"A prospective in silico analysis of interdisciplinary and interobserver spatial variability in post-operative target delineation of high-risk oral cavity cancers: Does physician specialty matter?","volume":"12","author":"Ng","year":"2018","journal-title":"Clin. Transl. Radiat. Oncol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1016\/j.ijrobp.2006.01.014","article-title":"Variability of gross tumor volume delineation in head-and-neck cancer using CT and PET\/CT fusion","volume":"65","author":"Riegel","year":"2006","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."}],"container-title":["Cancers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-6694\/17\/12\/1935\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:49:41Z","timestamp":1760032181000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-6694\/17\/12\/1935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,10]]},"references-count":42,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["cancers17121935"],"URL":"https:\/\/doi.org\/10.3390\/cancers17121935","relation":{},"ISSN":["2072-6694"],"issn-type":[{"value":"2072-6694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,10]]}}}