{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T22:37:52Z","timestamp":1793313472263,"version":"4.5.0"},"reference-count":16,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,9,12]],"date-time":"2025-09-12T00:00:00Z","timestamp":1757635200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,9,12]],"date-time":"2025-09-12T00:00:00Z","timestamp":1757635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004721","name":"The University of Tokyo","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004721","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Jpn J Radiol"],"published-print":{"date-parts":[[2026,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>To assess and compare the accuracy and legitimacy of multimodal large language models (LLMs) on the Japan Diagnostic Radiology Board Examination (JDRBE).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>The dataset comprised questions from JDRBE 2021, 2023, and 2024, with ground-truth answers established through consensus among multiple board-certified diagnostic radiologists. Questions without associated images and those lacking unanimous agreement on answers were excluded. Eight LLMs were evaluated: GPT-4 Turbo, GPT-4o, GPT-4.5, GPT-4.1, o3, o4-mini, Claude 3.7 Sonnet, and Gemini 2.5 Pro. Each model was evaluated under two conditions: with inputting images (vision) and without (text-only). Performance differences between the conditions were assessed using McNemar\u2019s exact test. Two diagnostic radiologists (with 2 and 18\u00a0years of experience) independently rated the legitimacy of responses from four models (GPT-4 Turbo, Claude 3.7 Sonnet, o3, and Gemini 2.5 Pro) using a five-point Likert scale, blinded to model identity. Legitimacy scores were analyzed using Friedman\u2019s test, followed by pairwise Wilcoxon signed-rank tests with Holm correction.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The dataset included 233 questions. Under the vision condition, o3 achieved the highest accuracy at 72%, followed by o4-mini (70%) and Gemini 2.5 Pro (70%). Under the text-only condition, o3 topped the list with an accuracy of 67%. Addition of image input significantly improved the accuracy of two models (Gemini 2.5 Pro and GPT-4.5), but not the others. Both o3 and Gemini 2.5 Pro received significantly higher legitimacy scores than GPT-4 Turbo and Claude 3.7 Sonnet from both raters.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Recent multimodal LLMs, particularly o3 and Gemini 2.5 Pro, have demonstrated remarkable progress on JDRBE questions, reflecting their rapid evolution in diagnostic radiology.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Secondary abstract<\/jats:title>\n                    <jats:p>Eight multimodal large language models were evaluated on the Japan Diagnostic Radiology Board Examination. OpenAI\u2019s o3 and Google DeepMind\u2019s Gemini 2.5 Pro achieved high accuracy rates (72% and 70%) and received good legitimacy scores from human raters, demonstrating steady progress.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11604-025-01861-y","type":"journal-article","created":{"date-parts":[[2025,9,12]],"date-time":"2025-09-12T04:22:48Z","timestamp":1757650968000},"page":"209-217","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Assessing accuracy and legitimacy of multimodal large language models on Japan Diagnostic Radiology Board Examination"],"prefix":"10.1007","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1564-0513","authenticated-orcid":false,"given":"Yuichiro","family":"Hirano","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6365-6531","authenticated-orcid":false,"given":"Soichiro","family":"Miki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7688-3075","authenticated-orcid":false,"given":"Yosuke","family":"Yamagishi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7496-1651","authenticated-orcid":false,"given":"Shouhei","family":"Hanaoka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9498-7501","authenticated-orcid":false,"given":"Takahiro","family":"Nakao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4222-4569","authenticated-orcid":false,"given":"Tomohiro","family":"Kikuchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6962-6704","authenticated-orcid":false,"given":"Yuta","family":"Nakamura","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6471-9936","authenticated-orcid":false,"given":"Yukihiro","family":"Nomura","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1924-5468","authenticated-orcid":false,"given":"Takeharu","family":"Yoshikawa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1180-2629","authenticated-orcid":false,"given":"Osamu","family":"Abe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,9,12]]},"reference":[{"issue":"1","key":"1861_CR1","doi-asserted-by":"publisher","first-page":"1372","DOI":"10.1186\/s12909-024-06309-x","volume":"24","author":"Y Chen","year":"2024","unstructured":"Chen Y, Huang X, Yang F, Lin H, Lin H, Zheng Z, et al. Performance of ChatGPT and Bard on the medical licensing examinations varies across different cultures: a comparison study. BMC Med Educ. 2024;24(1):1372.","journal-title":"BMC Med Educ"},{"issue":"1","key":"1861_CR2","doi-asserted-by":"publisher","DOI":"10.2196\/63129","volume":"10","author":"Y Miyazaki","year":"2024","unstructured":"Miyazaki Y, Hata M, Omori H, Hirashima A, Nakagawa Y, Eto M, et al. Performance of ChatGPT-4o on the Japanese Medical Licensing Examination: Evalution of accuracy in text-only and image-based questions. JMIR Med Educ. 2024;10(1): e63129.","journal-title":"JMIR Med Educ"},{"issue":"9","key":"1861_CR3","doi-asserted-by":"publisher","first-page":"543","DOI":"10.3390\/info15090543","volume":"15","author":"M Kipp","year":"2024","unstructured":"Kipp M. From GPT-3.5 to GPT-4.O: A leap in AI\u2019s medical exam performance. Information (Basel). 2024;15(9):543.","journal-title":"Information (Basel)."},{"issue":"8","key":"1861_CR4","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1007\/s11604-024-01561-z","volume":"42","author":"Y Hirano","year":"2024","unstructured":"Hirano Y, Hanaoka S, Nakao T, Miki S, Kikuchi T, Nakamura Y, et al. Gpt-4 turbo with vision fails to outperform text-only GPT-4 turbo in the Japan Diagnostic Radiology Board Examination. Jpn J Radiol. 2024;42(8):918\u201326.","journal-title":"Jpn J Radiol"},{"issue":"11","key":"1861_CR5","doi-asserted-by":"publisher","first-page":"1352","DOI":"10.1007\/s11604-024-01622-3","volume":"42","author":"Y Hirano","year":"2024","unstructured":"Hirano Y, Hanaoka S, Nakao T, Miki S, Kikuchi T, Nakamura Y, et al. No improvement found with GPT-4o: results of additional experiments in the Japan Diagnostic Radiology Board Examination. Jpn J Radiol. 2024;42(11):1352\u20133.","journal-title":"Jpn J Radiol"},{"issue":"5","key":"1861_CR6","doi-asserted-by":"publisher","first-page":"2394","DOI":"10.1016\/j.acra.2024.10.035","volume":"32","author":"T Nakaura","year":"2025","unstructured":"Nakaura T, Yoshida N, Kobayashi N, Nagayama Y, Uetani H, Kidoh M, et al. Performance of multimodal large language models in Japanese Diagnostic Radiology Board Examinations (2021\u20132023). Acad Radiol. 2025;32(5):2394\u2013401.","journal-title":"Acad Radiol"},{"issue":"2","key":"1861_CR7","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1007\/s12149-024-01992-8","volume":"39","author":"T Watanabe","year":"2025","unstructured":"Watanabe T, Baba A, Fukuda T, Watanabe K, Woo J, Ojiri H. Role of visual information in multimodal large language model performance: an evaluation using the Japanese nuclear medicine board examination. Ann Nucl Med. 2025;39(2):217\u201324.","journal-title":"Ann Nucl Med"},{"issue":"12","key":"1861_CR8","doi-asserted-by":"publisher","first-page":"1399","DOI":"10.1007\/s11604-024-01634-z","volume":"42","author":"R Kurokawa","year":"2024","unstructured":"Kurokawa R, Ohizumi Y, Kanzawa J, Kurokawa M, Sonoda Y, Nakamura Y, et al. Diagnostic performances of Claude 3 Opus and Claude 3.5 Sonnet from patient history and key images in Radiology\u2019s \u201cDiagnosis Please\u201d cases. Jpn J Radiol. 2024;42(12):1399\u2013402.","journal-title":"Jpn J Radiol"},{"issue":"12","key":"1861_CR9","doi-asserted-by":"publisher","first-page":"1392","DOI":"10.1007\/s11604-024-01633-0","volume":"42","author":"T Oura","year":"2024","unstructured":"Oura T, Tatekawa H, Horiuchi D, Matsushita S, Takita H, Atsukawa N, et al. Diagnostic accuracy of vision-language models on Japanese diagnostic radiology, nuclear medicine, and interventional radiology specialty board examinations. Jpn J Radiol. 2024;42(12):1392\u20138.","journal-title":"Jpn J Radiol"},{"issue":"11","key":"1861_CR10","doi-asserted-by":"publisher","first-page":"4365","DOI":"10.1016\/j.acra.2024.09.005","volume":"31","author":"MS Be\u015fler","year":"2024","unstructured":"Be\u015fler MS, Oleaga L, Junquero V, Merino C. Evaluating GPT-4o\u2019s performance in the official European Board of radiology exam: A comprehensive assessment. Acad Radiol. 2024;31(11):4365\u201371.","journal-title":"Acad Radiol"},{"issue":"9","key":"1861_CR11","doi-asserted-by":"publisher","first-page":"1080","DOI":"10.1007\/s11604-024-01574-8","volume":"42","author":"MS Be\u015fler","year":"2024","unstructured":"Be\u015fler MS. The accuracy of large language models in RANZCR\u2019s clinical radiology exam sample questions. Jpn J Radiol. 2024;42(9):1080.","journal-title":"Jpn J Radiol"},{"key":"1861_CR12","unstructured":"Reasoning models [Internet]. [cited 2025 May 23]. Available from: https:\/\/platform.openai.com\/docs\/guides\/reasoning?api-mode=responses"},{"key":"1861_CR13","unstructured":"Gemini 2.5 Pro [Internet]. Google DeepMind. [cited 2025 May 23]. Available from: https:\/\/deepmind.google\/models\/gemini\/pro\/"},{"key":"1861_CR14","unstructured":"Claude 3.7 Sonnet and Claude Code [Internet]. [cited 2025 May 21]. Available from: https:\/\/www.anthropic.com\/news\/claude-3-7-sonnet"},{"issue":"1","key":"1861_CR15","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159\u201374.","journal-title":"Biometrics"},{"issue":"1","key":"1861_CR16","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232411","volume":"310","author":"D Li","year":"2024","unstructured":"Li D, Gupta K, Bhaduri M, Sathiadoss P, Bhatnagar S, Chong J. Comparing GPT-3.5 and GPT-4 accuracy and drift in radiology diagnosis please cases. Radiology. 2024;310(1): e232411.","journal-title":"Radiology"}],"container-title":["Japanese Journal of Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11604-025-01861-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11604-025-01861-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11604-025-01861-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T05:33:36Z","timestamp":1767591216000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11604-025-01861-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,12]]},"references-count":16,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1]]}},"alternative-id":["1861"],"URL":"https:\/\/doi.org\/10.1007\/s11604-025-01861-y","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2025.06.23.25329534","asserted-by":"object"}]},"ISSN":["1867-1071","1867-108X"],"issn-type":[{"value":"1867-1071","type":"print"},{"value":"1867-108X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,12]]},"assertion":[{"value":"12 June 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 August 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflicts of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This study did not directly involve human subjects. All data used in this study have been anonymized and are devoid of any information that could identify individuals, and these are available online to all JRS members. Therefore, approval from the Institutional Review Board was waived.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}