{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T01:23:09Z","timestamp":1775438589102,"version":"3.50.1"},"reference-count":27,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T00:00:00Z","timestamp":1775174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives. Abdominal computed tomography (CT) radiology reports contain critical information for abdominal aortic aneurysm (AAA) management, including aneurysm presence, size, rupture status, and prior repair. However, this information is often embedded within lengthy, heterogeneous reports, making manual extraction inefficient. We evaluated the performance of multiple large language models (LLMs) for automated extraction of AAA-related findings from radiology reports. Methods. We retrospectively analyzed 500 abdominal CT reports mentioning AAA from an urban academic health system (2020\u20132024). Ground truth labels were established by manual review. Four open-source LLMs (Qwen2.5-7B-Instruct, Llama3-Med42-8B, GPT-OSS-20B, and MedGemma-27B-text-it) were evaluated for extraction of aneurysm presence, size, morphology, rupture status, impending rupture, and prior aortic repair. Model outputs were compared with ground truth using exact-match accuracy, and inter-model agreement was assessed using Fleiss\u2019 kappa. Reasoning traces were examined to characterize correct and incorrect model behavior. Results. Accuracy for identifying AAA presence ranged from 0.90 to 0.95 (\u03ba = 0.851), and prior aortic repair from 0.90 to 0.97 (\u03ba = 0.793). Accuracy for aneurysm size ranged from 0.67 to 0.88 (\u03ba = 0.340), with low \u03ba\u2019s due to class imbalance or dimension misselection. Rupture and impending rupture were identified with accuracies exceeding 0.90 across models, though agreement was lower (\u03ba = 0.485 and 0.589), reflecting low event prevalence. Larger models (GPT-OSS-20B, MedGemma-27B) generally outperformed smaller models. Reasoning analysis revealed strengths in measurement prioritization but recurrent errors, including dimension misselection, over-inference of prior repair, and conservative classification of rupture-related findings. Conclusions. LLMs can accurately extract clinically relevant AAA information from radiology reports with interpretable reasoning, with larger and medically trained models outperforming smaller or general-purpose models. Performance varies by task and model, underscoring the need for careful validation and human-in-the-loop deployment in clinical settings.<\/jats:p>","DOI":"10.3390\/diagnostics16071083","type":"journal-article","created":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T00:19:32Z","timestamp":1775434772000},"page":"1083","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Leveraging Large Language Models for Automated Extraction of Abdominal Aortic Aneurysm Features from Radiology Reports"],"prefix":"10.3390","volume":"16","author":[{"given":"Praneel","family":"Mukherjee","sequence":"first","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3448-0730","authenticated-orcid":false,"given":"Ryan C.","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"},{"name":"Renaissance School of Medicine, Stony Brook University, Stony Brook, NY 11794, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2231-0172","authenticated-orcid":false,"given":"Roham","family":"Hadidchi","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6471-4569","authenticated-orcid":false,"given":"Sonya","family":"Henry","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Coard","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Davis","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yossef","family":"Rubinov","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2263-609X","authenticated-orcid":false,"given":"Ha","family":"Nguyen-Luong","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leah","family":"Katz","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6403-2827","authenticated-orcid":false,"given":"Tim Q.","family":"Duong","sequence":"additional","affiliation":[{"name":"Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,4,3]]},"reference":[{"key":"#cr-split#-ref_1.1","unstructured":"Expert Panel on Vascular Imaging, Wang, D.S., Shen, J., Majdalany, B.S., Khaja, M.S., Bhatti, S., Ferencik, M., Ganguli, S., Gunn, A.J., and Heitner, J.F. (2023). ACR Appropriateness Criteria"},{"key":"#cr-split#-ref_1.2","doi-asserted-by":"crossref","unstructured":"(R) Pulsatile Abdominal Mass, Suspected Abdominal Aortic Aneurysm: 2023 Update. J. Am. Coll. Radiol., 20, S513-S520.","DOI":"10.1016\/j.jacr.2023.08.010"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Expert Panel on Vascular Imaging, Lee, Y.J., Aghayev, A., Azene, E.M., Bhatti, S., Ewell, J.C., Hedgire, S.S., Kendi, A.T., Kim, E.S.H., and Kirsch, D.S. (2024). ACR Appropriateness Criteria(R) Screening for Abdominal Aortic Aneurysm. J. Am. Coll. Radiol., 21, S286\u2013S291.","DOI":"10.1016\/j.jacr.2024.02.027"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.jvs.2017.10.044","article-title":"The Society for Vascular Surgery practice guidelines on the care of patients with an abdominal aortic aneurysm","volume":"67","author":"Chaikof","year":"2018","journal-title":"J. Vasc. Surg."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2211","DOI":"10.1001\/jama.2019.18928","article-title":"Screening for Abdominal Aortic Aneurysm: US Preventive Services Task Force Recommendation Statement","volume":"322","author":"Force","year":"2019","journal-title":"Jama"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.ejvs.2018.09.020","article-title":"Editor\u2019s Choice\u2014European Society for Vascular Surgery (ESVS) 2019 Clinical Practice Guidelines on the Management of Abdominal Aorto-iliac Artery Aneurysms","volume":"57","author":"Wanhainen","year":"2019","journal-title":"Eur. J. Vasc. Endovasc. Surg."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2848","DOI":"10.1161\/CIRCULATIONAHA.110.014902","article-title":"Predictors of abdominal aortic aneurysm sac enlargement after endovascular repair","volume":"123","author":"Schanzer","year":"2011","journal-title":"Circulation"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1016\/j.jacc.2022.04.066","article-title":"Repair of Abdominal Aortic Aneurysms: JACC Focus Seminar, Part 1","volume":"80","author":"Hensley","year":"2022","journal-title":"J. Am. Coll. Cardiol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Martinelli, O., Cuozzo, S., Miceli, F., Gattuso, R., D\u2019Andrea, V., Sapienza, P., and Bellini, M.I. (2023). Elective Endovascular Aneurysm Repair (EVAR) for the Treatment of Infrarenal Abdominal Aortic Aneurysms of 5.0\u20135.5 cm: Differences between Men and Women. J. Clin. Med., 12.","DOI":"10.3390\/jcm12134364"},{"key":"ref_9","first-page":"827","article-title":"Utilizing natural language processing to improve abdominal aortic aneurysm surveillance","volume":"82","author":"DiLosa","year":"2025","journal-title":"J. Vasc. Surg."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"922","DOI":"10.1016\/j.jvs.2022.10.034","article-title":"Successful implementation of a nurse-navigator-run program using natural language processing identifying patients with an abdominal aortic aneurysm","volume":"77","author":"Boitano","year":"2023","journal-title":"J. Vasc. Surg."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e312","DOI":"10.1016\/j.jvs.2025.03.458","article-title":"Large language models accurately extract aortic information from abdominal imaging reports in a large, real-world database","volume":"81","author":"Flanagan","year":"2025","journal-title":"J. Vasc. Surg."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1016\/j.jvs.2020.12.090","article-title":"Validation of natural language processing to determine the presence and size of abdominal aortic aneurysms in a large integrated health system","volume":"74","author":"McLenon","year":"2021","journal-title":"J. Vasc. Surg."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., and McDermott, M. (2019). Publicly Available Clinical BERT Embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop: 2019, Association for Computational Linguistics.","DOI":"10.18653\/v1\/W19-1909"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"26388","DOI":"10.1038\/s41598-025-11870-6","article-title":"Natural Language Processing framework for identifying abdominal aortic aneurysm repairs using unstructured electronic health records","volume":"15","author":"Thompson","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.avsg.2024.09.059","article-title":"Enhancing Aortic Aneurysm Surveillance: Transformer Natural Language Processing for Flagging and Measuring in Radiology Reports","volume":"110","author":"Kartsonis","year":"2025","journal-title":"Ann. Vasc. Surg."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1016\/j.jacr.2025.10.005","article-title":"Use of Large Language Models on Radiology Reports: A Scoping Review","volume":"23","author":"Lee","year":"2025","journal-title":"J. Am. Coll. Radiol."},{"key":"ref_17","unstructured":"Christophe, C., Kanithi, P., Raha, T., Khan, S., and Pimentel, M. (2024). Med42-v2: A Suite of Clinical LLMs. arXiv."},{"key":"ref_18","unstructured":"Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., and Lau, C. (2025). MedGemma Technical Report. arXiv."},{"key":"ref_19","unstructured":"Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., and Huang, F. (2024). Qwen2 Technical Report. arXiv."},{"key":"ref_20","unstructured":"Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R.K., Bai, Y., and Baker, B. (2025). gpt-oss-120b & gpt-oss-20b Model Card. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3885","DOI":"10.1007\/s00261-025-04810-5","article-title":"Identifying abdominal aortic aneurysm size and presence using Natural Language Processing of radiology reports: A systematic review and meta-analysis","volume":"50","author":"Sajjadi","year":"2025","journal-title":"Abdom. Radiol."},{"key":"ref_22","first-page":"249","article-title":"Identifying Abdominal Aortic Aneurysm Cases and Controls using Natural Language Processing of Radiology Reports","volume":"2013","author":"Sohn","year":"2013","journal-title":"AMIA Jt. Summits Transl. Sci. Proc."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"9799","DOI":"10.1038\/s41467-025-64769-1","article-title":"Quantifying the reasoning abilities of LLMs on clinical cases","volume":"16","author":"Qiu","year":"2025","journal-title":"Nat. Commun."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"973","DOI":"10.1007\/s10278-025-01530-6","article-title":"High-Performance Prompting for LLM Extraction of Compression Fracture Findings from Radiology Reports","volume":"39","author":"Kanani","year":"2025","journal-title":"J. Imaging Inform. Med."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e243774","DOI":"10.1148\/radiol.243774","article-title":"Comparison of a Specialized Large Language Model with GPT-4o for CT and MRI Radiology Report Summarization","volume":"316","author":"Zheng","year":"2025","journal-title":"Radiology"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1038\/s41746-024-01029-4","article-title":"Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs","volume":"7","author":"Wang","year":"2024","journal-title":"NPJ Digit. Med."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/7\/1083\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T00:30:47Z","timestamp":1775435447000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/7\/1083"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,3]]},"references-count":27,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["diagnostics16071083"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16071083","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,3]]}}}