{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T10:45:00Z","timestamp":1775817900419,"version":"3.50.1"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T00:00:00Z","timestamp":1775779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Oncol."],"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to evaluate the ability of GPT-4o and Gemini 2.5 Pro to extract and assign PI-RADS v2.1 score from free-text prostate MRI reports, and compare their performance with human readers of varied experience.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Three radiologists with differing levels of experience (resident, fellow, expert) independently reviewed the reports and assigned PI-RADS v2.1 scores. The same reports were processed through prompts with the GPT-4o and Gemini 2.5 Pro. Inter-rater agreement was evaluated using Gwet\u2019s AC1 coefficient, and the diagnostic performance was assessed using sensitivity, specificity, and area under the receiver operating characteristic curve (AUC).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Inter-rater agreement between human experts was highest between the expert and fellow (Gwet\u2019s AC1\u00a0=\u00a00.68, 95% CI 0.61-0.75), which was significantly higher than between two LLMs (Gwet\u2019s AC1\u00a0=\u00a00.52, 95% CI 0.44-0.59, P\u00a0=\u00a00.004). The agreement between expert and GPT (Gwet\u2019s AC1\u00a0=\u00a00.42, 95% CI 0.34-0.51) was lower than between expert and Gemini (Gwet\u2019s AC1\u00a0=\u00a00.49, 95% CI 0.41-0.57, P\u00a0=\u00a00.17). The AUCs for resident, fellow, and expert readers were 0.81 (95% CI 0.76-0.87), 0.86 (95% CI 0.81-0.91), and 0.89 (95% CI 0.85-0.93), and for GPT and Gemini were 0.85 (95% CI 0.81-0.90) and 0.84 (95% CI 0.80-0.89), respectively.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>LLMs demonstrated promising performance in assigning PI-RADS scores from free-text prostate MRI reports, with accuracy and agreement approaching that of general radiologists; however, they are not yet ready to replace expert interpretation in high-stakes clinical settings. Nevertheless, these findings support its potential as a supplementary tool for report standardization and trainee education.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fonc.2026.1743096","type":"journal-article","created":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T09:45:01Z","timestamp":1775814301000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluation of large language models for PI-RADS score extraction from free-text prostate MRI reports: a comparative study with human readers"],"prefix":"10.3389","volume":"16","author":[{"given":"Jing","family":"Wen","sequence":"first","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yifan","family":"Qian","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xun","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yifan","family":"Cui","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingjing","family":"Hao","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xin","family":"Yuan","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhihan","family":"Tan","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zihao","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenhao","family":"Meng","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging, Jiangsu Medical College","place":["Yancheng, China"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2026,4,10]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.3322\/caac.21660","article-title":"Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries","volume":"71","author":"Sung","year":"2021","journal-title":"CA Cancer J Clin"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eururo.2019.02.033","article-title":"Prostate imaging reporting and data system version 2.1: 2019 update of prostate imaging reporting and data system version 2","volume":"76","author":"Turkbey","year":"2019","journal-title":"Eur Urol"},{"key":"B3","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.eururo.2015.08.052","article-title":"PI-RADS prostate imaging \u2013 reporting and data system: 2015, version 2","volume":"69","author":"Weinreb","year":"2016","journal-title":"Eur Urol"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-011-2377-y","article-title":"ESUR prostate MR guidelines 2012","volume":"22","author":"Barentsz","year":"2012","journal-title":"Eur Radiol"},{"key":"B5","doi-asserted-by":"publisher","first-page":"793","DOI":"10.1148\/radiol.2016152542","article-title":"Interobserver reproducibility of the PI-RADS version 2 lexicon: a multicenter study of six experienced prostate radiologists","volume":"280","author":"Rosenkrantz","year":"2016","journal-title":"Radiology"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2022.1013941","article-title":"Inter-reader agreement of the prostate imaging reporting and data system version v2.1 for detection of prostate cancer: a systematic review and meta-analysis","volume":"12","author":"Wen","year":"10","journal-title":"Front Oncol"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231593","article-title":"Quantitative evaluation of large language models to streamline radiology report impressions: a multimodal retrospective analysis","volume":"310","author":"Doshi","year":"2024","journal-title":"Radiology"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232640","article-title":"Monitoring patients with glioblastoma by using a large language model: accurate summarization of radiology reports with GPT-4","volume":"312","author":"Laukamp","year":"2024","journal-title":"Radiology"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"B10","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"Mesk\u00f3","year":"2023","journal-title":"NPJ Digit Med"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra2301725","article-title":"The current and future state of AI interpretation of medical images","volume":"388","author":"Rajpurkar","year":"2023","journal-title":"N Engl J Med"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.242575","article-title":"Generative large language models trained for detecting errors in radiology reports","volume":"315","author":"Sun","year":"2025","journal-title":"Radiology"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231362","article-title":"Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer","volume":"308","author":"Fink","year":"2023","journal-title":"Radiology"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232133","article-title":"BI-RADS category assignments by GPT-3.5, GPT-4, and Google Bard: a multilanguage study","volume":"311","author":"Cozzi","year":"2024","journal-title":"Radiology"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.2214\/AJR.24.31696","article-title":"Lung cancer staging using chest CT and FDG PET\/CT free-text reports: comparison among three ChatGPT large language models and six human readers of varying experience","volume":"223","author":"Lee","year":"2024","journal-title":"Am J Roentgenol"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-024-11032-8","article-title":"Comparative analysis of GPT-4-based ChatGPT\u2019s diagnostic performance with radiologists using real-world radiology reports of brain tumors","volume":"35","author":"Mitsuyama","year":"2025","journal-title":"Eur Radiol"},{"key":"B17","unstructured":"Google AI Studio"},{"key":"B18","unstructured":"OpenAI"},{"key":"B19","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1177\/1536867X1801800408","article-title":"Implementing a general framework for assessing interrater agreement in Stata","volume":"18","author":"Klein","year":"2018","journal-title":"Stata J"},{"key":"B20","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","year":"2008","journal-title":"Br J Math Stat Psychol"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1093\/bjr\/tqae236","article-title":"Assessing the performance of ChatGPT and Bard\/Gemini against radiologists for prostate imaging-reporting and data system classification based on prostate multiparametric MRI text reports","volume":"98","author":"Lee","year":"2025","journal-title":"Br J Radiol"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230163","article-title":"ChatGPT and other large language models are double-edged swords","volume":"307","author":"Shen","year":"2023","journal-title":"Radiology"},{"key":"B23","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1002\/jmri.26882","article-title":"Interobserver agreement of PI-RADS v. 2 lexicon among radiologists with different levels of experience","volume":"51","author":"Mussi","year":"2020","journal-title":"J Magn Reson Imaging JMRI"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1002\/jmri.26220","article-title":"Interreader agreement of PI-RADS v. 2 in assessing prostate cancer with multiparametric MRI: a study using whole-mount histology as the standard of reference","volume":"49","author":"Girometti","year":"2019","journal-title":"J Magn Reson Imaging JMRI"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1056\/nejmsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1016\/j.diii.2024.04.003","article-title":"ChatGPT in radiology: a systematic review of performance, pitfalls, and future perspectives","volume":"105","author":"Keshavarz","year":"2024","journal-title":"Diagn Interv Imaging"},{"key":"B27","doi-asserted-by":"publisher","first-page":"7866","DOI":"10.1038\/s41467-025-62385-7","article-title":"Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data","volume":"16","author":"Wu","year":"2025","journal-title":"Nat Commun"},{"key":"B28","unstructured":"MedGemma | Health AI Developer Foundations"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2506.12186","article-title":"MRI-CORE: A foundation model for magnetic resonance imaging","author":"Dong","year":"2025"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1002\/pros.24699","article-title":"Performance of large language models on benign prostatic hyperplasia frequently asked questions","volume":"84","author":"Zhang","year":"2024","journal-title":"Prostate"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.1002\/pros.24814","article-title":"Quality of chatbot information related to benign prostatic hyperplasia","volume":"85","author":"Warren","year":"2025","journal-title":"Prostate"}],"container-title":["Frontiers in Oncology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2026.1743096\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T09:45:06Z","timestamp":1775814306000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fonc.2026.1743096\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,10]]},"references-count":31,"alternative-id":["10.3389\/fonc.2026.1743096"],"URL":"https:\/\/doi.org\/10.3389\/fonc.2026.1743096","relation":{},"ISSN":["2234-943X"],"issn-type":[{"value":"2234-943X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,10]]},"article-number":"1743096"}}