{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T00:49:44Z","timestamp":1791938984163,"version":"4.3.4"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T00:00:00Z","timestamp":1781827200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T00:00:00Z","timestamp":1781827200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To evaluate whether GPT-4.1 and Llama 3.3 70B, large language models (LLMs) assessed in zero-shot, baseline configurations, detect and categorize clinically consequential errors across types that range from pattern-based to reasoning-dependent.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>\n                      Two hundred fifty-six radiology reports encompassing CT (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u2009104), MRI (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u2009104), and X-ray (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u200948) studies across multiple anatomical regions were retrospectively analyzed. For each original report, four variants (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u20091024) were generated, each incorporating one of four predefined error types: E1, anatomical mislabeling that could cause wrong-site actions; E2, physiologically impossible or nonsensical findings; E3, diagnostic inconsistencies that affect staging or diagnosis; E4, inappropriate recommendations. The evaluated models were GPT\u20114.1 04-14) and Llama 3.3 70B, both used without domain-specific training or prompt optimization to assess baseline model performance.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Model performance revealed a systematic hierarchy governed by error type and imaging modality. Physiologically impossible errors (E2) showed the lowest performance: 46.2% (CT) and 33.7% (MRI) for GPT-4.1, compared with 32.7% and 25.0% for Llama 3.3, respectively. Overall success for GPT-4.1 on E2 was 16.3% (CT), 8.7% (MRI), and 12.5% (X-ray). Mislabeling errors (E1) were detected in 49.0% by GPT\u20114.1 and 33.7% by Llama 3.3 for MRI. Best performance occurred for inappropriate recommendations (E4), with GPT\u20114.1 achieving 85.4% detection in X-ray with high classification accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The evaluation framework and benchmark dataset provide a methodology for assessing LLM performance on clinically significant errors. Applied to GPT-4.1 and Llama 3.3 70B in zero-shot settings, the framework reveals a performance gap between pattern-based and reasoning-dependent error detection that warrants investigation across additional models and optimization strategies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Key Points<\/jats:title>\n                    <jats:p>\n                      <jats:bold>\n                        <jats:italic>Question<\/jats:italic>\n                      <\/jats:bold>\n                      <jats:italic>LLMs are increasingly used for quality assurance of radiology reports, but whether their linguistic competence translates into the detection of clinically significant errors remains unclear<\/jats:italic>\n                      .\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:bold>\n                        <jats:italic>Findings<\/jats:italic>\n                      <\/jats:bold>\n                      <jats:italic>Error detection was type-dependent, both GPT-4.1 and Llama 3.3 70B performed poorly on physiologic and anatomical errors, but better on inappropriate recommendations<\/jats:italic>\n                      .\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:bold>\n                        <jats:italic>Clinical relevance<\/jats:italic>\n                      <\/jats:bold>\n                      <jats:italic>LLMs failed to detect most clinically consequential errors in radiology reports, especially physiologically impossible statements that trained radiologists would rapidly identify<\/jats:italic>\n                      .\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s00330-026-12697-z","type":"journal-article","created":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T11:01:11Z","timestamp":1781866871000},"page":"8149-8159","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["GPT-4.1 and Llama 3.3 70 fail to detect clinically relevant errors in radiology reports in zero-shot evaluation"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7237-711X","authenticated-orcid":false,"given":"Tugba","family":"Akinci D\u2019Antonoli","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa C.","family":"Adams","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jannik","family":"L\u00fcbberstedt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus M.","family":"Graf","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian J.","family":"Mertens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Busch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sebastian","family":"Ziegelmayer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marcus R.","family":"Makowski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keno","family":"Bressem","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ina","family":"Luiken","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,6,19]]},"reference":[{"key":"12697_CR1","doi-asserted-by":"publisher","first-page":"111032","DOI":"10.1016\/j.ejrad.2023.111032","volume":"167","author":"\u00d6 Kasalak","year":"2023","unstructured":"Kasalak \u00d6, Alnahwi H, Toxopeus R, Pennings JP, Yakar D, Kwee TC (2023) Work overload and diagnostic errors in radiology. Eur J Radiol 167:111032. https:\/\/doi.org\/10.1016\/j.ejrad.2023.111032","journal-title":"Eur J Radiol"},{"key":"12697_CR2","doi-asserted-by":"publisher","first-page":"e232756","DOI":"10.1148\/radiol.232756","volume":"310","author":"R Bhayana","year":"2024","unstructured":"Bhayana R (2024) Chatbots and large language models in radiology: a practical primer for clinical and research applications. Radiology 310:e232756. https:\/\/doi.org\/10.1148\/radiol.232756","journal-title":"Radiology"},{"key":"12697_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3390\/jcm14103285","volume":"14","author":"M Tordjman","year":"2025","unstructured":"Tordjman M, Bolger I, Yuce M et al (2025) Large language models in cancer imaging: applications and future perspectives. J Clin Med 14:1\u201315. https:\/\/doi.org\/10.3390\/jcm14103285","journal-title":"J Clin Med"},{"key":"12697_CR4","doi-asserted-by":"publisher","first-page":"e240528","DOI":"10.1148\/radiol.240528","volume":"315","author":"C Bluethgen","year":"2025","unstructured":"Bluethgen C, van Veen D, Zakka C et al (2025) Best practices for large language models in radiology. Radiology 315:e240528. https:\/\/doi.org\/10.1148\/radiol.240528","journal-title":"Radiology"},{"key":"12697_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-024-01219-0]","volume":"7","author":"D Reichenpfader","year":"2024","unstructured":"Reichenpfader D, M\u00fcller H, Denecke K (2024) A scoping review of large language model based approaches for information extraction from radiology reports. npj Digit Med 7:1\u201312. https:\/\/doi.org\/10.1038\/s41746-024-01219-0]","journal-title":"npj Digit Med"},{"key":"12697_CR6","doi-asserted-by":"publisher","first-page":"2589","DOI":"10.1007\/s00330-024-11107-6]","volume":"35","author":"F Busch","year":"2025","unstructured":"Busch F, Hoffmann L, dos Santos DP et al (2025) Large language models for structured reporting in radiology: past, present, and future. Eur Radiol 35:2589\u20132602. https:\/\/doi.org\/10.1007\/s00330-024-11107-6]","journal-title":"Eur Radiol"},{"key":"12697_CR7","unstructured":"Wang Z, Luo X, Jiang X, Li D, Qiu L (2024) LLM-RadJudge: achieving radiologist-level evaluation for X-ray report generation. Preprint at https:\/\/arxiv.org\/abs\/2404.00998"},{"key":"12697_CR8","doi-asserted-by":"publisher","first-page":"126","DOI":"10.3348\/kjr.2023.0997","volume":"25","author":"S Kim","year":"2024","unstructured":"Kim S, Lee C-K, Kim S-S (2024) Large language models: a guide for radiologists. Korean J Radiol 25:126\u2013133. https:\/\/doi.org\/10.3348\/kjr.2023.0997","journal-title":"Korean J Radiol"},{"key":"12697_CR9","doi-asserted-by":"publisher","first-page":"e230364","DOI":"10.1148\/ryai.230364","volume":"6","author":"B Le Guellec","year":"2024","unstructured":"Le Guellec B, Lef\u00e8vre A, Geay C et al (2024) Performance of an open-source large language model in extracting information from free-text radiology reports. Radio Artif Intell 6:e230364. https:\/\/doi.org\/10.1148\/ryai.230364","journal-title":"Radio Artif Intell"},{"key":"12697_CR10","doi-asserted-by":"publisher","unstructured":"Takita H, Walston SL, Mitsuyama Y, Watanabe K, Ishimaru S, Ueda D (2025) Comparative performance of large language models in structuring head CT radiology reports: multi-institutional validation study in Japan. Jpn J Radiol. https:\/\/doi.org\/10.1007\/s11604-025-01799-1","DOI":"10.1007\/s11604-025-01799-1"},{"key":"12697_CR11","doi-asserted-by":"publisher","unstructured":"Schmidt RA, Seah JCY, Cao K, Lim L, Lim W, Yeung J (2024) Generative large language models for detection of speech recognition errors in radiology reports. Radiol Artif Intell. https:\/\/doi.org\/10.1148\/ryai.230205","DOI":"10.1148\/ryai.230205"},{"key":"12697_CR12","doi-asserted-by":"publisher","first-page":"e242575","DOI":"10.1148\/radiol.242575","volume":"315","author":"C Sun","year":"2025","unstructured":"Sun C, Teichman K, Zhou Y et al (2025) Generative large language models trained for detecting errors in radiology reports. Radiology 315:e242575. https:\/\/doi.org\/10.1148\/radiol.242575","journal-title":"Radiology"},{"key":"12697_CR13","doi-asserted-by":"publisher","unstructured":"Alabed S, Anderson A, Maiter A et al (2026) Large language models for simplifying radiology reports: a systematic review and meta-analysis of patient, public, and clinician evaluations. The Lancet Digital Health 8:2. https:\/\/doi.org\/10.1016\/j.landig.2025.100960","DOI":"10.1016\/j.landig.2025.100960"},{"key":"12697_CR14","doi-asserted-by":"publisher","first-page":"e240739","DOI":"10.1148\/ryai.240739","volume":"7","author":"T Akinci D\u2019Antonoli","year":"2025","unstructured":"Akinci D\u2019Antonoli T, Tejani AS, Khosravi B et al (2025) Cybersecurity threats and mitigation strategies for large language models in health care. Radio Artif Intell 7:e240739. https:\/\/doi.org\/10.1148\/ryai.240739","journal-title":"Radio Artif Intell"},{"key":"12697_CR15","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240701","volume":"314","author":"S Kim","year":"2025","unstructured":"Kim S, Kim D, Shin HJ et al (2025) Large-scale validation of the feasibility of GPT-4 as a proofreading tool for head CT reports. Radiology 314:e240701. https:\/\/doi.org\/10.1148\/radiol.240701","journal-title":"Radiology"},{"key":"12697_CR16","doi-asserted-by":"crossref","unstructured":"Toma A, Xie R, Palayew S, Lawler PR, Wang B (2024) WangLab at MEDIQA-CORR 2024: optimized LLM-based programs for medical error detection and correction. Association for Computational Linguistics, Mexico City, pp 616\u2013623","DOI":"10.18653\/v1\/2024.clinicalnlp-1.59"},{"key":"12697_CR17","doi-asserted-by":"publisher","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","volume":"34","author":"K Jeblick","year":"2024","unstructured":"Jeblick K, Schachtner B, Dexl J et al (2024) ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur Radiol 34:2817\u20132825. https:\/\/doi.org\/10.1007\/s00330-023-10213-1","journal-title":"Eur Radiol"},{"key":"12697_CR18","doi-asserted-by":"crossref","unstructured":"Ostmeier S, Xu J, Chen Z et al (2024) GREEN: generative radiology report evaluation and error notation. Association for Computational Linguistics, Florida, pp 374\u2013390. https:\/\/aclanthology.org","DOI":"10.18653\/v1\/2024.findings-emnlp.21"},{"key":"12697_CR19","doi-asserted-by":"publisher","unstructured":"Sloan P, Clatworthy P, Simpson E, Mirmehdi M (2025) Automated radiology report generation: a review of recent advances. IEEE Rev Biomed Eng 18:368\u2013387. https:\/\/doi.org\/10.1109\/RBME.2024.3408456","DOI":"10.1109\/RBME.2024.3408456"},{"key":"12697_CR20","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1038\/s41562-024-02046-9","volume":"9","author":"X Luo","year":"2025","unstructured":"Luo X, Rechardt A, Sun G et al (2025) Large language models surpass human experts in predicting neuroscience results. Nat Hum Behav 9:305\u2013315. https:\/\/doi.org\/10.1038\/s41562-024-02046-9","journal-title":"Nat Hum Behav"},{"key":"12697_CR21","doi-asserted-by":"publisher","first-page":"3498","DOI":"10.1016\/j.csbj.2024.09.031","volume":"23","author":"OA Sarumi","year":"2024","unstructured":"Sarumi OA, Heider D (2024) Large language models and their applications in bioinformatics. Comput Struct Biotechnol J 23:3498\u20133505. https:\/\/doi.org\/10.1016\/j.csbj.2024.09.031","journal-title":"Comput Struct Biotechnol J"},{"key":"12697_CR22","doi-asserted-by":"publisher","first-page":"57","DOI":"10.3390\/informatics11030057","volume":"11","author":"ZA Nazi","year":"2024","unstructured":"Nazi ZA, Peng W (2024) Large language models in healthcare and medical domain: a review. Informatics 11:57. https:\/\/doi.org\/10.3390\/informatics11030057","journal-title":"Informatics"},{"key":"12697_CR23","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232714","volume":"311","author":"RJ Gertz","year":"2024","unstructured":"Gertz RJ, Dratsch T, Bunck AC et al (2024) Potential of GPT-4 for detecting errors in radiology reports: implications for reporting accuracy. Radiology 311:e232714. https:\/\/doi.org\/10.1148\/radiol.232714","journal-title":"Radiology"},{"key":"12697_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/20552076251326019","volume":"11","author":"Z Xu","year":"2025","unstructured":"Xu Z, Ye J, Luo W et al (2025) An assessment of ChatGPT in error detection for thyroid ultrasound reports: a comparative study with ultrasound physicians. DIGITAL HEALTH 11:1\u20139. https:\/\/doi.org\/10.1177\/20552076251326019","journal-title":"DIGITAL HEALTH"},{"key":"12697_CR25","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","volume":"630","author":"S Farquhar","year":"2024","unstructured":"Farquhar S, Kossen J, Kuhn L, Gal Y (2024) Detecting hallucinations in large language models using semantic entropy. Nature 630:625\u2013630. https:\/\/doi.org\/10.1038\/s41586-024-07421-0","journal-title":"Nature"},{"key":"12697_CR26","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1038\/s43856-025-01021-3","volume":"5","author":"M Omar","year":"2025","unstructured":"Omar M, Sorin V, Collins JD et al (2025) Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Commun Med (Lond) 5:330. https:\/\/doi.org\/10.1038\/s43856-025-01021-3","journal-title":"Commun Med (Lond)"},{"key":"12697_CR27","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1038\/s41746-024-01239-w","volume":"7","author":"G Zhang","year":"2024","unstructured":"Zhang G, Jin Q, Zhou Y et al (2024) Closing the gap between open source and commercial large language models for medical evidence summarization. npj Digit Med 7:239. https:\/\/doi.org\/10.1038\/s41746-024-01239-w","journal-title":"npj Digit Med"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-026-12697-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-026-12697-z","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-026-12697-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T00:03:32Z","timestamp":1791936212000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-026-12697-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,19]]},"references-count":27,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2026,11]]}},"alternative-id":["12697"],"URL":"https:\/\/doi.org\/10.1007\/s00330-026-12697-z","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,19]]},"assertion":[{"value":"20 February 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 April 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 May 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 June 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The scientific guarantor of this publication is Ina Luiken.","order":1,"name":"Ethics","label":"Guarantor","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"Tugba Akinci D\u2019Antonoli, Lisa Adams and Keno Bressem are members of the European Radiology Editorial Board, and as such, they have not participated in the review or selection processes for this article. The remaining authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.","order":2,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"No complex statistical methods were necessary for this paper.","order":3,"name":"Ethics","label":"Statistics and biometry","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"Written informed consent was not required for this study because of its retrospective nature.","order":4,"name":"Ethics","label":"Informed consent","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"Institutional Review Board approval was obtained.","order":5,"name":"Ethics","label":"Ethical approval","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"No overlap.","order":6,"name":"Ethics","label":"Study subjects or cohorts overlap","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"order":7,"name":"Ethics","label":"Methodology","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}}]}}