{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:45:20Z","timestamp":1792788320596,"version":"4.5.0"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T00:00:00Z","timestamp":1768953600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T00:00:00Z","timestamp":1768953600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["5T32HL007185-47"],"award-info":[{"award-number":["5T32HL007185-47"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The purpose of this study was to investigate the performance of LLMs in radiology numerical tasks and perform a comprehensive error analysis. We defined six tasks: extracting (1) minimum T-score from DEXA report, (2) maximum common bile duct (CBD) diameter from ultrasound report, and (3) maximum lung nodule size from CT report, and judging (1) presence of a highly hypermetabolic region on a PET report, (2) whether a patient is osteoporotic based on a DEXA report, and (3) whether a patient has a dilated CBD based on an ultrasound report. Reports were extracted from the MIMIC III and our institution\u2019s databases, and the ground truths were extracted manually. The models used were Llama 3.1 8b, DeepSeek R1 distilled Llama 8b, OpenAI o1-mini, and OpenAI GPT-5-mini. We manually reviewed all incorrect outputs and performed a comprehensive error analysis. In extraction tasks, while Llama showed relatively variable results (ranging 86%\u201398.7%) across tasks, other models performed consistently well (accuracies\u2009&gt;\u200995%). In judgment tasks, the lowest accuracies of Llama, DeepSeek distilled Llama, o1-mini, and GPT-5-mini were 62.0%, 91.7%, 91.7%, and 99.0%, respectively, while o1-mini and GPT-5-mini did reach 100% performance in detecting osteoporosis. We found no mathematical errors in the outputs of o1-mini and GPT-5-mini. Answer-only output format significantly reduced performance in Llama and DeepSeek distilled Llama but not in o1-mini or GPT-5-mini. To conclude,\u00a0reinforcement learning (RL) reasoning models perform consistently well in radiology numerical tasks and show no mathematical errors. Simpler non-RL reasoning models may also achieve acceptable performance depending on the task.<\/jats:p>","DOI":"10.1007\/s10278-025-01824-9","type":"journal-article","created":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T16:10:53Z","timestamp":1769011853000},"page":"4828-4839","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Large Language Models in Radiologic Numerical Tasks: A Thorough Evaluation and Error Analysis"],"prefix":"10.1007","volume":"39","author":[{"given":"Ali","family":"Nowroozi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masha","family":"Bondarenko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian","family":"Serapio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tician","family":"Schnitzler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sukhmanjit S.","family":"Brar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6733-7551","authenticated-orcid":false,"given":"Jae Ho","family":"Sohn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,1,21]]},"reference":[{"key":"1824_CR1","doi-asserted-by":"crossref","unstructured":"Hager P, Jungmann F, Holland R, Bhagat K, Hubrecht I, Knauer M, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med. 2024 Sept;30(9):2613\u201322.","DOI":"10.1038\/s41591-024-03097-1"},{"key":"1824_CR2","doi-asserted-by":"crossref","unstructured":"Park J, Oh K, Han K, Lee YH. Patient-centered radiology reports with generative artificial intelligence: adding value to radiology reporting. Sci Rep. 2024 June 8;14(1):13218.","DOI":"10.1038\/s41598-024-63824-z"},{"key":"1824_CR3","unstructured":"Gliadkovskaya A. Some doctors are using ChatGPT to assist with clinical decisions. Is it safe? [Internet]. 2024 [cited 2025 Nov 25]. Available from: https:\/\/www.fiercehealthcare.com\/special-reports\/some-doctors-are-using-public-generative-ai-tools-chatgpt-clinical-decisions-it"},{"key":"1824_CR4","doi-asserted-by":"crossref","unstructured":"Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med Inform Decis Mak. 2025 Mar 7;25(1):117.","DOI":"10.1186\/s12911-025-02954-4"},{"key":"1824_CR5","doi-asserted-by":"crossref","unstructured":"Bicknell BT, Butler D, Whalen S, Ricks J, Dixon CJ, Clark AB, et al. ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis. JMIR Med Educ. 2024 Nov 6;10(1):e63430.","DOI":"10.2196\/63430"},{"key":"1824_CR6","doi-asserted-by":"crossref","unstructured":"Testolin A. Can Neural Networks Do Arithmetic? A Survey on the Elementary Numerical Skills of State-of-the-Art Deep Learning Models. Appl Sci. 2024 Jan;14(2):744.","DOI":"10.3390\/app14020744"},{"key":"1824_CR7","doi-asserted-by":"crossref","unstructured":"Akhtar M, Shankarampeta A, Gupta V, Patil A, Cocarascu O, Simperl E. Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data. In: Bouamor H, Pino J, Bali K, editors. Findings of the Association for Computational Linguistics: EMNLP 2023 [Internet]. Singapore: Association for Computational Linguistics; 2023 [cited 2024 Sept 2]. p. 15391\u2013405. Available from: https:\/\/aclanthology.org\/2023.findings-emnlp.1028","DOI":"10.18653\/v1\/2023.findings-emnlp.1028"},{"key":"1824_CR8","doi-asserted-by":"crossref","unstructured":"Goodell AJ, Chu SN, Rouholiman D, Chu LF. Large language model agents can use tools to perform clinical calculations. NPJ Digit Med. 2025 Mar 17;8:163.","DOI":"10.1038\/s41746-025-01475-8"},{"key":"1824_CR9","doi-asserted-by":"crossref","unstructured":"Wu Q, Li H, Wang Y, Bai Y, Wu Y, Yu X, et al. Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study. JMIR Med Inform. 2024 July 17;12(1):e55799.","DOI":"10.2196\/55799"},{"key":"1824_CR10","unstructured":"LiveBench [Internet]. [cited 2025 Mar 19]. Available from: https:\/\/livebench.ai\/#\/"},{"key":"1824_CR11","unstructured":"OpenRouter [Internet]. [cited 2025 Mar 19]. LLM Rankings. Available from: https:\/\/openrouter.ai"},{"key":"1824_CR12","unstructured":"Pricing [Internet]. [cited 2025 Mar 19]. Available from: https:\/\/openai.com\/api\/pricing\/"},{"key":"1824_CR13","unstructured":"Pricing [Internet]. [cited 2025 Mar 19]. Available from: https:\/\/www.anthropic.com\/pricing#anthropic-api"},{"key":"1824_CR14","doi-asserted-by":"crossref","unstructured":"Patel D, Raut G, Zimlichman E, Cheetirala SN, Nadkarni GN, Glicksberg BS, et al. Evaluating prompt engineering on GPT-3.5\u2019s performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4. Sci Rep. 2024 July 28;14:17341.","DOI":"10.1038\/s41598-024-66933-x"},{"key":"1824_CR15","unstructured":"Khandekar N, Jin Q, Xiong G, Dunn S, Applebaum SS, Anwar Z, et al. MedCalc-Bench: Evaluating Large Language Models for Medical Calculations [Internet]. arXiv; 2024 [cited 2025 Jan 15]. Available from: http:\/\/arxiv.org\/abs\/2406.12036"},{"key":"1824_CR16","doi-asserted-by":"crossref","unstructured":"Levin C, Orkaby B, Kerner E, Saban M. Can large language models assist with pediatric dosing accuracy? Pediatr Res. 2025 Mar 8;1\u20136.","DOI":"10.1038\/s41390-025-03980-8"},{"key":"1824_CR17","unstructured":"Malghan AR. Evaluating Computational Accuracy of Large Language Models in Numerical Reasoning Tasks for Healthcare Applications [Internet]. arXiv; 2025 [cited 2025 May 13]. Available from: http:\/\/arxiv.org\/abs\/2501.13936"},{"key":"1824_CR18","doi-asserted-by":"crossref","unstructured":"Le Guellec B, Lef\u00e8vre A, Geay C, Shorten L, Bruge C, Hacein-Bey L, et al. Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports. Radiol Artif Intell. 2024 July;6(4):e230364.","DOI":"10.1148\/ryai.230364"},{"key":"1824_CR19","doi-asserted-by":"crossref","unstructured":"Reichenpfader D, M\u00fcller H, Denecke K. A scoping review of large language model based approaches for information extraction from radiology reports. Npj Digit Med. 2024 Aug 24;7(1):1\u201312.","DOI":"10.1038\/s41746-024-01219-0"},{"key":"1824_CR20","doi-asserted-by":"crossref","unstructured":"Johnson AEW, Pollard TJ, Shen L, Lehman L wei H, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016 May 24;3(1):160035.","DOI":"10.1038\/sdata.2016.35"},{"key":"1824_CR21","unstructured":"meta-llama\/Meta-Llama-3.1\u20138B-Instruct \u00b7 Hugging Face [Internet]. 2024 [cited 2024 Sept 3]. Available from: https:\/\/huggingface.co\/meta-llama\/Meta-Llama-3.1-8B-Instruct"},{"key":"1824_CR22","unstructured":"DeepSeek-AI, Guo D, Yang D, Zhang H, Song J, Zhang R, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [Internet]. arXiv; 2025 [cited 2025 Feb 18]. Available from: http:\/\/arxiv.org\/abs\/2501.12948"},{"key":"1824_CR23","unstructured":"deepseek-ai\/DeepSeek-R1-Distill-Llama-8B \u00b7 Hugging Face [Internet]. 2025 [cited 2025 Feb 18]. Available from: https:\/\/huggingface.co\/deepseek-ai\/DeepSeek-R1-Distill-Llama-8B"},{"key":"1824_CR24","unstructured":"Hugging Face \u2013 The AI community building the future. [Internet]. 2024 [cited 2024 Sept 3]. Available from: https:\/\/huggingface.co\/"},{"key":"1824_CR25","unstructured":"Transformers [Internet]. [cited 2024 Sept 3]. Available from: https:\/\/huggingface.co\/docs\/transformers\/en\/index"},{"key":"1824_CR26","doi-asserted-by":"crossref","unstructured":"Wilson EB. Probable Inference, the Law of Succession, and Statistical Inference. J Am Stat Assoc. 1927 June 1;22(158):209\u201312.","DOI":"10.1080\/01621459.1927.10502953"},{"key":"1824_CR27","doi-asserted-by":"crossref","unstructured":"Wang L, Chen X, Deng X, Wen H, You M, Liu W, et al. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. Npj Digit Med. 2024 Feb 20;7(1):1\u20139.","DOI":"10.1038\/s41746-024-01029-4"},{"key":"1824_CR28","unstructured":"Hwang Y, Kim Y, Koo J, Kang T, Bae H, Jung K. LLMs can be easily Confused by Instructional Distractions [Internet]. arXiv; 2025 [cited 2025 May 14]. Available from: http:\/\/arxiv.org\/abs\/2502.04362"},{"key":"1824_CR29","unstructured":"Levy M, Jacoby A, Goldberg Y. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models [Internet]. arXiv; 2024 [cited 2025 May 14]. Available from: http:\/\/arxiv.org\/abs\/2402.14848"},{"key":"1824_CR30","unstructured":"Tam ZR, Wu CK, Tsai YL, Lin CY, Lee H yi, Chen YN. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models [Internet]. arXiv; 2024 [cited 2025 May 14]. Available from: http:\/\/arxiv.org\/abs\/2408.02442"},{"key":"1824_CR31","doi-asserted-by":"crossref","unstructured":"Bhasuran B, Jin Q, Xie Y, Yang C, Hanna K, Costa J, et al. Preliminary analysis of the impact of lab results on large language model generated differential diagnoses. Npj Digit Med. 2025 Mar 18;8(1):1\u201315.","DOI":"10.1038\/s41746-025-01556-8"},{"key":"1824_CR32","unstructured":"MDCalc [Internet]. [cited 2025 June 13]. MDCalc - Medical calculators, equations, scores, and guidelines. Available from: https:\/\/www.mdcalc.com\/"},{"key":"1824_CR33","doi-asserted-by":"crossref","unstructured":"Goodell AJ, Chu SN, Rouholiman D, Chu LF. Large language model agents can use tools to perform clinical calculations. Npj Digit Med. 2025 Mar 17;8(1):163.","DOI":"10.1038\/s41746-025-01475-8"},{"key":"1824_CR34","doi-asserted-by":"crossref","unstructured":"Dobrocky T, Grunder L, Breiding PS, Branca M, Limacher A, Mosimann PJ, et al. Assessing Spinal Cerebrospinal Fluid Leaks in Spontaneous Intracranial Hypotension With a Scoring System Based on Brain Magnetic Resonance Imaging Findings. JAMA Neurol. 2019 May 1;76(5):580\u20137.","DOI":"10.1001\/jamaneurol.2018.4921"}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-025-01824-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-025-01824-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-025-01824-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:17:25Z","timestamp":1792786645000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-025-01824-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,21]]},"references-count":34,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["1824"],"URL":"https:\/\/doi.org\/10.1007\/s10278-025-01824-9","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,21]]},"assertion":[{"value":"8 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 December 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 January 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This study was approved by our institution\u2019s IRB (approval number 17\u201322317).","order":1,"name":"Ethics","label":"Ethics Approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","label":"Competing Interests","group":{"name":"EthicsHeading","label":"Declarations"}}]}}