{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T10:46:55Z","timestamp":1793357215792,"version":"4.5.0"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T00:00:00Z","timestamp":1747180800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T00:00:00Z","timestamp":1747180800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"the Accreditation Organization for Management of Radiologic Imaging"},{"name":"Iida Sangyo Co., Ltd."}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Jpn J Radiol"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>To compare the diagnostic performance of three proprietary large language models (LLMs)\u2014Claude, GPT, and Gemini\u2014in structuring free-text Japanese radiology reports for intracranial hemorrhage and skull fractures, and to assess the impact of three different prompting approaches on model accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>In this retrospective study, head CT reports from the Japan Medical Imaging Database between 2018 and 2023 were collected. Two board-certified radiologists established the ground truth regarding intracranial hemorrhage and skull fractures through independent review and consensus. Each radiology report was analyzed by three LLMs using three prompting strategies\u2014Standard, Chain of Thought, and Self Consistency prompting. Diagnostic performance (accuracy, precision, recall, and F1-score) was calculated for each LLM\u2013prompt combination and compared using McNemar\u2019s tests with Bonferroni correction. Misclassified cases underwent qualitative error analysis.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      A total of 3949 head CT reports from 3949 patients (mean age 59\u2009\u00b1\u200925\u00a0years, 56.2% male) were enrolled. Across all institutions, 856 patients (21.6%) had intracranial hemorrhage and 264 patients (6.6%) had skull fractures. All nine LLM\u2013prompt combinations achieved very high accuracy. Claude demonstrated significantly higher accuracy for intracranial hemorrhage than GPT and Gemini, and also outperformed Gemini for skull fractures (\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.0001). Gemini\u2019s performance improved notably with Chain of Thought prompting. Error analysis revealed common challenges including ambiguous phrases and findings unrelated to intracranial hemorrhage or skull fractures, underscoring the importance of careful prompt design.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>All three proprietary LLMs exhibited strong performance in structuring free-text head CT reports for intracranial hemorrhage and skull fractures. While the choice of prompting method influenced accuracy, all models demonstrated robust potential for clinical and research applications. Future work should refine the prompts and validate these approaches in prospective, multilingual settings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11604-025-01799-1","type":"journal-article","created":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T07:35:04Z","timestamp":1747208104000},"page":"1445-1455","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Comparative performance of large language models in structuring head CT radiology reports: multi-institutional validation study in Japan"],"prefix":"10.1007","volume":"43","author":[{"given":"Hirotaka","family":"Takita","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shannon L.","family":"Walston","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasuhito","family":"Mitsuyama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ko","family":"Watanabe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shoya","family":"Ishimaru","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3878-3616","authenticated-orcid":false,"given":"Daiju","family":"Ueda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,5,14]]},"reference":[{"key":"1799_CR1","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1227\/NEU.0000000000001432","volume":"80","author":"N Carney","year":"2017","unstructured":"Carney N, Totten AM, O\u2019Reilly C, Ullman JS, Hawryluk GWJ, Bell MJ, et al. Guidelines for the management of severe traumatic brain injury, Fourth Edition. Neurosurgery. 2017;80:6\u201315.","journal-title":"Neurosurgery"},{"key":"1799_CR2","doi-asserted-by":"publisher","first-page":"1391","DOI":"10.1016\/S0140-6736(00)04561-X","volume":"357","author":"IG Stiell","year":"2001","unstructured":"Stiell IG, Wells GA, Vandemheen K, Clement C, Lesiuk H, Laupacis A, et al. The Canadian CT head rule for patients with minor head injury. Lancet. 2001;357:1391\u20136.","journal-title":"Lancet"},{"key":"1799_CR3","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1056\/NEJM200007133430204","volume":"343","author":"MJ Haydel","year":"2000","unstructured":"Haydel MJ, Preston CA, Mills TJ, Luber S, Blaudeau E, DeBlieux PM. Indications for computed tomography in patients with minor head injury. N Engl J Med. 2000;343:100\u20135.","journal-title":"N Engl J Med"},{"key":"1799_CR4","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1148\/radiol.2523081992","volume":"252","author":"CE Kahn Jr","year":"2009","unstructured":"Kahn CE Jr, Langlotz CP, Burnside ES, Carrino JA, Channin DS, Hovsepian DM, et al. Toward best practices in radiology reporting. Radiology. 2009;252:852\u20136.","journal-title":"Radiology"},{"key":"1799_CR5","doi-asserted-by":"publisher","first-page":"240","DOI":"10.1148\/radiol.12121502","volume":"267","author":"DB Larson","year":"2013","unstructured":"Larson DB, Towbin AJ, Pryor RM, Donnelly LF. Improving consistency in radiology reporting through the use of department-wide standardized structured reporting. Radiology. 2013;267:240\u201350.","journal-title":"Radiology"},{"key":"1799_CR6","doi-asserted-by":"publisher","unstructured":"Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. arXiv [Internet]. 2020; https:\/\/doi.org\/10.48550\/arXiv.2005.14165","DOI":"10.48550\/arXiv.2005.14165"},{"key":"1799_CR7","doi-asserted-by":"publisher","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv [Internet]. 2019; https:\/\/doi.org\/10.48550\/arXiv.1810.04805","DOI":"10.48550\/arXiv.1810.04805"},{"key":"1799_CR8","doi-asserted-by":"publisher","unstructured":"Takita H, Kabata D, Walston SL, Tatekawa H, Saito K, Tsujimoto Y, et al. Diagnostic performance comparison between generative AI and physicians: a systematic review and meta-analysis [Internet]. medRxiv. 2024. https:\/\/doi.org\/10.1101\/2024.01.20.24301563","DOI":"10.1101\/2024.01.20.24301563"},{"key":"1799_CR9","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-024-11032-8","author":"Y Mitsuyama","year":"2024","unstructured":"Mitsuyama Y, Tatekawa H, Takita H, Sasaki F, Tashiro A, Oue S, et al. Comparative analysis of GPT-4-based ChatGPT\u2019s diagnostic performance with radiologists using real-world radiology reports of brain tumors. Eur Radiol. 2024. https:\/\/doi.org\/10.1007\/s00330-024-11032-8.","journal-title":"Eur Radiol"},{"key":"1799_CR10","doi-asserted-by":"publisher","first-page":"845","DOI":"10.1148\/radiol.2017171115","volume":"286","author":"MC Chen","year":"2018","unstructured":"Chen MC, Ball RL, Yang L, Moradzadeh N, Chapman BE, Larson DB, et al. Deep learning to classify radiology free-text reports. Radiology. 2018;286:845\u201352.","journal-title":"Radiology"},{"key":"1799_CR11","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.233117","author":"R Bhayana","year":"2024","unstructured":"Bhayana R, Nanda B, Dehkharghanian T, Deng Y, Bhambra N, Elias G, et al. Large language models for automated synoptic reports and resectability categorization in pancreatic cancer. Radiology. 2024. https:\/\/doi.org\/10.1148\/radiol.233117.","journal-title":"Radiology"},{"key":"1799_CR12","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231040","volume":"308","author":"D Ueda","year":"2023","unstructured":"Ueda D, Mitsuyama Y, Takita H, Horiuchi D, Walston SL, Tatekawa H, et al. ChatGPT\u2019s diagnostic performance from patient history and imaging findings on the diagnosis please quizzes. Radiology. 2023;308: e231040.","journal-title":"Radiology"},{"key":"1799_CR13","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1007\/s00234-023-03252-4","volume":"66","author":"D Horiuchi","year":"2024","unstructured":"Horiuchi D, Tatekawa H, Shimono T, Walston SL, Takita H, Matsushita S, et al. Accuracy of ChatGPT generated diagnosis from patient\u2019s medical history and imaging findings in neuroradiology cases. Neuroradiology. 2024;66:73\u20139.","journal-title":"Neuroradiology"},{"key":"1799_CR14","doi-asserted-by":"publisher","first-page":"472","DOI":"10.1016\/j.jvir.2023.11.014","volume":"35","author":"F Sasaki","year":"2024","unstructured":"Sasaki F, Tatekawa H, Mitsuyama Y, Kageyama K, Jogo A, Yamamoto A, et al. Bridging language and stylistic barriers in IR standardized reporting: enhancing translation and structure using ChatGPT-4. J Vasc Interv Radiol. 2024;35:472-5.e1.","journal-title":"J Vasc Interv Radiol"},{"key":"1799_CR15","doi-asserted-by":"publisher","first-page":"506","DOI":"10.1007\/s00330-024-10902-5","volume":"35","author":"D Horiuchi","year":"2025","unstructured":"Horiuchi D, Tatekawa H, Oura T, Shimono T, Walston SL, Takita H, et al. ChatGPT\u2019s diagnostic performance based on textual vs. visual information compared to radiologists\u2019 diagnostic performance in musculoskeletal radiology. Eur Radiol. 2025;35:506\u201316.","journal-title":"Eur Radiol"},{"key":"1799_CR16","doi-asserted-by":"publisher","DOI":"10.1007\/s11604-024-01633-0","author":"T Oura","year":"2024","unstructured":"Oura T, Tatekawa H, Horiuchi D, Matsushita S, Takita H, Atsukawa N, et al. Diagnostic accuracy of vision-language models on Japanese diagnostic radiology, nuclear medicine, and interventional radiology specialty board examinations. Jpn J Radiol. 2024. https:\/\/doi.org\/10.1007\/s11604-024-01633-0.","journal-title":"Jpn J Radiol"},{"key":"1799_CR17","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1007\/s00062-024-01426-y","volume":"34","author":"D Horiuchi","year":"2024","unstructured":"Horiuchi D, Tatekawa H, Oura T, Oue S, Walston SL, Takita H, et al. Comparing the diagnostic performance of GPT-4-based ChatGPT, GPT-4V-based ChatGPT, and radiologists in challenging neuroradiology cases. Clin Neuroradiol. 2024;34:779\u201387.","journal-title":"Clin Neuroradiol"},{"key":"1799_CR18","doi-asserted-by":"publisher","unstructured":"Busch F, Hoffmann L, Dos Santos DP, Makowski MR, Saba L, Prucker P, et al. Large language models for structured reporting in radiology: past, present, and future. Eur Radiol [Internet]. 2024; https:\/\/doi.org\/10.1007\/s00330-024-11107-6","DOI":"10.1007\/s00330-024-11107-6"},{"key":"1799_CR19","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/s10140-019-01673-4","volume":"26","author":"P Pruitt","year":"2019","unstructured":"Pruitt P, Naidech A, Van Ornam J, Borczuk P, Thompson W. A natural language processing algorithm to extract characteristics of subdural hematoma from head CT reports. Emerg Radiol. 2019;26:301\u20136.","journal-title":"Emerg Radiol"},{"key":"1799_CR20","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1148\/radiol.2015151516","volume":"277","author":"PM Bossuyt","year":"2015","unstructured":"Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology. 2015;277:826\u201332.","journal-title":"Radiology"},{"key":"1799_CR21","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/978-981-16-6376-5_13","volume-title":"Epidemiologic research on real-world medical data in Japan","author":"D Kakihara","year":"2022","unstructured":"Kakihara D, Nishie A, Machitori A, Honda H. The Japan medical imaging database (J-MID). In: Epidemiologic research on real-world medical data in Japan. Springer Nature Singapore; 2022. p. 87\u201393."},{"key":"1799_CR22","doi-asserted-by":"publisher","unstructured":"Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models. arXiv [Internet]. 2023; https:\/\/doi.org\/10.48550\/arXiv.2201.11903","DOI":"10.48550\/arXiv.2201.11903"},{"key":"1799_CR23","doi-asserted-by":"publisher","unstructured":"Wang X, Wei J, Schuurmans D, et al. Self-consistency improves chain of thought reasoning in language models. arXiv [Internet]. 2023; https:\/\/doi.org\/10.48550\/arXiv.2203.11171","DOI":"10.48550\/arXiv.2203.11171"},{"key":"1799_CR24","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230725","author":"LC Adams","year":"2023","unstructured":"Adams LC, Truhn D, Busch F, Kader A, Niehues SM, Makowski MR, et al. Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: a multilingual feasibility study. Radiology. 2023. https:\/\/doi.org\/10.1148\/radiol.230725.","journal-title":"Radiology"},{"key":"1799_CR25","unstructured":"Model card Claude 3.5 [Internet]. [cited 2025 Feb 11]. https:\/\/www-cdn.anthropic.com\/fed9cc193a14b84131812372d8d5857f8f304c52\/Model_Card_Claude_3_Addendum.pdf"},{"key":"1799_CR26","volume":"16","author":"R Javan","year":"2024","unstructured":"Javan R, Kim T, Abdelmonem A, Ismail A, Jaamour F, Melnyk O, et al. Programming chatbots using natural language: generating cervical spine MRI impressions. Cureus. 2024;16: e69410.","journal-title":"Cureus"},{"key":"1799_CR27","doi-asserted-by":"publisher","first-page":"888","DOI":"10.1016\/j.acra.2024.09.041","volume":"32","author":"E Can","year":"2025","unstructured":"Can E, Uller W, Vogt K, Doppler MC, Busch F, Bayerl N, et al. Large language models for simplified interventional radiology reports: a comparative analysis. Acad Radiol. 2025;32:888\u201398.","journal-title":"Acad Radiol"},{"key":"1799_CR28","doi-asserted-by":"publisher","DOI":"10.2196\/55799","volume":"12","author":"Q Wu","year":"2024","unstructured":"Wu Q, Wu Q, Li H, Wang Y, Bai Y, Wu Y, et al. Evaluating large language models for automated reporting and data systems categorization: cross-sectional study. JMIR Med Inform. 2024;12: e55799.","journal-title":"JMIR Med Inform"},{"key":"1799_CR29","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s11604-023-01474-3","volume":"42","author":"D Ueda","year":"2023","unstructured":"Ueda D, Kakinuma T, Fujita S, Kamagata K, Fushimi Y, Ito R, et al. Fairness of artificial intelligence in healthcare: review and recommendations. Jpn J Radiol. 2023;42:3\u201315.","journal-title":"Jpn J Radiol"}],"container-title":["Japanese Journal of Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11604-025-01799-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11604-025-01799-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11604-025-01799-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,6]],"date-time":"2025-09-06T14:47:38Z","timestamp":1757170058000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11604-025-01799-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,14]]},"references-count":29,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["1799"],"URL":"https:\/\/doi.org\/10.1007\/s11604-025-01799-1","relation":{},"ISSN":["1867-1071","1867-108X"],"issn-type":[{"value":"1867-1071","type":"print"},{"value":"1867-108X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,14]]},"assertion":[{"value":"28 February 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 May 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 May 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The study protocol was approved by the Ethical Committee of Juntendo University Graduate School of Medicine and Osaka Metropolitan Universtiy, and the study was conducted in accordance with the Declaration of Helsinki.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"The requirement for informed consent was waived because, in the Japanese Medical Imaging Database, radiology reports are completely anonymized and there is no concern about identifying personal information.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}]}}