{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T20:03:46Z","timestamp":1786737826240,"version":"build-2736575974"},"reference-count":14,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"2","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"abstract":"<jats:sec>\n                    <jats:title>Objectives:<\/jats:title>\n                    <jats:p>This study aims to evaluate various large language models (LLMs) for their effectiveness in answering Japan Radiology Board Examination (JRBE).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods:<\/jats:title>\n                    <jats:p>\n                      A total of 315 examination questions from 2021 to 2023 JRBE were administered to 14 LLMs, comprising 7 open-source and 7 closed-source models. Each model processed the questions in their original Japanese language and after translating them into English by LLMs themselves. Performance metrics, including median scores, interquartile ranges (IQR),\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      -values, and correlation coefficients, were analyzed using Python.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>\n                      Closed-source models achieved a higher median correct response rate of 44.28% (IQR: 32.38%\u201353.02%) compared with open-source models at 29.52% (IQR: 24.76%\u201336.67%), exhibiting a 50.2% improvement (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      &lt; 0.001). Translating questions to English improved performance, with median scores increasing from 33.02% (IQR: 27.86%\u201343.65%) to 40.00% (IQR: 27.54%\u201346.27%), representing a 21.1% increase (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      = 0.005). A positive correlation was observed between the cost per 1M tokens and accuracy (Japanese: r = 0.623,\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      = 0.017; English: r = 0.613,\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      = 0.020). No significant correlation was found between model release dates and performance. Only the high-cost closed-source models, GPT-4 and Claude-3-Opus, surpassed the 60% passing threshold by Japanese Medical Specialty Board when using English translation. Among open-source models, LLama-3-70B demonstrated notable performance improvements compared with its predecessors.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions:<\/jats:title>\n                    <jats:p>Closed-source, high-end LLMs exhibit superior performance in JRBE, and translating questions into English by themselves further enhances their accuracy. There is a significant positive correlation between the cost of LLMs and their performance, whereas the release date does not significantly influence their performance.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1097\/rct.0000000000001807","type":"journal-article","created":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T20:00:06Z","timestamp":1762286406000},"page":"294-300","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Model Cost and Performance: A Comprehensive Analysis in the Context of the Japan Radiology Board Examination"],"prefix":"10.1097","volume":"50","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9010-0341","authenticated-orcid":false,"given":"Takeshi","family":"Nakaura","sequence":"first","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naoki","family":"Kobayashi","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaori","family":"Shiraishi","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naofumi","family":"Yoshida","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasunori","family":"Nagayama","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hiroyuki","family":"Uetani","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masafumi","family":"Kidoh","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seitaro","family":"Oda","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoshinori","family":"Funama","sequence":"additional","affiliation":[{"name":"Department of Medical Physics, Faculty of Life Sciences, Kumamoto University, Honjo 1-1-1, Kumamoto, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Toshinori","family":"Hirai","sequence":"additional","affiliation":[{"name":"Department of Diagnostic Radiology, Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2026,3]]},"reference":[{"key":"R1-20260814","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLoS Digital Health"},{"key":"R2-20260814","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment.","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"key":"R3-20260814","doi-asserted-by":"crossref","first-page":"e50965","DOI":"10.2196\/50965","article-title":"Comparison of the performance of GPT-3.5 and GPT-4 With that of medical students on the written German medical licensing examination: observational study","volume":"10","author":"Meyer","year":"2024","journal-title":"JMIR Med Educ"},{"key":"R4-20260814","doi-asserted-by":"crossref","first-page":"e34673","DOI":"10.1097\/MD.0000000000034673","article-title":"ChatGPT performance in the medical specialty exam: an observational study","volume":"102","author":"Oztermeli","year":"2023","journal-title":"Medicine (Baltimore)"},{"key":"R5-20260814","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1007\/s11604-023-01487-y","article-title":"Preliminary assessment of automated radiology report generation with generative pre-trained transformers: comparing results to radiologist-generated reports","volume":"42","author":"Nakaura","year":"2024","journal-title":"Jpn J Radiol"},{"key":"R8-20260814","doi-asserted-by":"crossref","first-page":"AIdbp2300092","DOI":"10.1056\/AIdbp2300092","article-title":"Benchmarking open-source large language models, GPT-4 and Claude 2 on multiple-choice questions in nephrology","volume":"1","author":"Wu","year":"2024","journal-title":"NEJM AI"},{"key":"R9-20260814","doi-asserted-by":"crossref","first-page":"e2417641","DOI":"10.1001\/jamanetworkopen.2024.17641","article-title":"Performance of large language models on medical oncology examination questions","volume":"7","author":"Longwell","year":"2024","journal-title":"JAMA Netw Open"},{"key":"R10-20260814","doi-asserted-by":"crossref","first-page":"e241191","DOI":"10.1148\/radiol.241191","article-title":"Llama 3 challenges proprietary state-of-the-art large language models in radiology board\u2013style examination questions","volume":"312","author":"Adams","year":"2024","journal-title":"Radiology"},{"key":"R11-20260814","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"key":"R12-20260814","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1007\/s11604-023-01491-2","article-title":"Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society","volume":"42","author":"Toyama","year":"2024","journal-title":"Jpn J Radiol"},{"key":"R13-20260814","doi-asserted-by":"crossref","first-page":"918","DOI":"10.1007\/s11604-024-01561-z","article-title":"GPT-4 Turbo with vision fails to outperform text-only GPT-4 Turbo in the Japan Diagnostic Radiology Board Examination","volume":"42","author":"Hirano","year":"2024","journal-title":"Jpn J Radiol"},{"key":"R15-20260814","doi-asserted-by":"crossref","first-page":"96","DOI":"10.15212\/RADSCI-2023-0007","article-title":"Application and technology of an open source AI large language model in the medical field","volume":"2","author":"Zhang","year":"2023","journal-title":"Radiology Science"},{"key":"R16-20260814","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1207\/s15516709cog1402_1","article-title":"Finding structure in time","volume":"14","author":"Elman","year":"1990","journal-title":"Cogn Sci"},{"key":"R18-20260814","doi-asserted-by":"crossref","first-page":"2394","DOI":"10.1016\/j.acra.2024.10.035","article-title":"Performance of multimodal large language models in Japanese Diagnostic Radiology Board Examinations (2021-2023)","volume":"32","author":"Nakaura","year":"2025","journal-title":"Acad Radiol"}],"container-title":["Journal of Computer Assisted Tomography"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.1097\/RCT.0000000000001807","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T19:00:09Z","timestamp":1786734009000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.1097\/RCT.0000000000001807"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,3]]},"references-count":14,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.1097\/rct.0000000000001807","relation":{},"ISSN":["1532-3145"],"issn-type":[{"value":"1532-3145","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,3]]},"assertion":[{"value":"2025-04-21","name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-28","name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}}]}}