{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T22:30:22Z","timestamp":1788301822073,"version":"build-2803163510"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language model (LLM) proofreaders for radiology reports generate many false positives (FPs) due to the low prevalence of errors.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to determine whether an optimized LLM framework could improve both precision and cost-efficiency without compromising error detection capability.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n In this retrospective study, 1000 radiology reports (radiography, ultrasonography, computed tomography, and magnetic resonance imaging; 250 each) were sampled from the Medical Information Mart for Intensive Care III database. Two public chest radiography corpora (CheXpert and Open-i) served as external test sets. Three LLM frameworks were evaluated: single-prompt detector (framework 1); report extractor plus single-prompt detector (framework 2); and extractor, detector, and FP verifier (framework 3). Precision for each framework was assessed using positive predictive value (PPV) and detected errors per 1000 reports. Overall efficiency was estimated using model inference costs and reviewer labor costs.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n PPV increased from 0.063 (95% CI 0.036\u20100.101) in framework 1 to 0.079 (95% CI 0.049\u20100.118) in framework 2 and 0.159 (95% CI 0.090\u20100.252) in framework 3 (\n P<\/jats:italic>\n <.001). Despite improved PPV, detected errors remained stable (detected errors per 1000 reports: 12\u201014). Human review burden decreased from 192 to 88 reports. Framework 3 also reduced model inference costs to US $5.57 per 1000 reports (vs US $9.72 and US $6.85 for frameworks 1 and 2; 42.6% and 18.5% reductions, respectively). External validation confirmed similar improvements. Qualitative analysis revealed that remaining FPs in framework 3 were largely confined to cases requiring deep clinical context (clinically equivalent rephrasing: 53%; unsupported discrepancy assertions: 43%). By eliminating structural FPs (eg, section mismatches and lexical errors: 0%), the framework effectively shifted the quality assurance burden to a smaller set of ambiguous cases, enabling a targeted human-in-the-loop workflow.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The multipass LLM improved the precision and cost-efficiency of radiology report error detection in real-world, low-error prevalence settings. The framework demonstrates the feasibility of synergistic artificial intelligence\u2013radiologist collaboration and provides a cost-effective and scalable approach to artificial intelligence\u2013assisted quality assurance in both radiological practice and research.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/87368","type":"journal-article","created":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T15:55:08Z","timestamp":1776786908000},"page":"e87368-e87368","source":"Crossref","is-referenced-by-count":1,"title":["Improving Radiology Report Error Detection Using a Multipass Large Language Model: Framework Development and Validation"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6908-4324","authenticated-orcid":false,"given":"Songsoo","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8320-0838","authenticated-orcid":false,"given":"Seungtae","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7293-3518","authenticated-orcid":false,"given":"See Young","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9020-8195","authenticated-orcid":false,"given":"Joonho","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9349-9960","authenticated-orcid":false,"given":"Keechan","family":"Kan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2523-1380","authenticated-orcid":false,"given":"Hyunji","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1635-8376","authenticated-orcid":false,"given":"Dukyong","family":"Yoon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,4]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232714","article-title":"Potential of GPT-4 for detecting errors in radiology reports: implications for reporting accuracy","volume":"311","author":"Gertz","journal-title":"Radiology"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240844","article-title":"Large language models as an inexpensive and effective extra set of eyes in radiology reporting","volume":"311","author":"Forman","journal-title":"Radiology"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240701","article-title":"Large-scale validation of the feasibility of GPT-4 as a proofreading tool for head CT reports","volume":"314","author":"Kim","journal-title":"Radiology"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1148\/radiol.213101","article-title":"Advancing artificial intelligence to meet breast imaging needs","volume":"303","author":"Philpotts","journal-title":"Radiology"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232030","article-title":"Environmental sustainability and AI in radiology: a double-edged sword","volume":"310","author":"Doo","journal-title":"Radiology"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01550-0","article-title":"Enhancing diagnostic capability with multi-agents conversational large language models","volume":"8","author":"Chen","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01533-1","article-title":"Medical foundation large language models for comprehensive text analysis and beyond","volume":"8","author":"Xie","journal-title":"NPJ Digit Med"},{"key":"R8","unstructured":"Cemri M Pan MZ Yang S et al. Why do multi-agent LLM systems fail? Presented at: 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Track on Datasets and Benchmarks; Dec 2-7, 2025; San Diego, CA. URL: https:\/\/openreview.net\/pdf?id=fAjbYBmonr [Accessed 09-05-2026]"},{"key":"R9","doi-asserted-by":"crossref","unstructured":"Moll J Fay L Azhar A et al. Structuring radiology reports: challenging LLMs with lightweight models. Presented at: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing; Dec 4-9, 2025; Suzhou, China. [doi: 10.18653\/v1\/2025.emnlp-main.392]","DOI":"10.18653\/v1\/2025.emnlp-main.392"},{"issue":"8","key":"R10","doi-asserted-by":"publisher","first-page":"4549","DOI":"10.1007\/s00330-025-11438-y","article-title":"Large language models for error detection in radiology reports: a comparative analysis between closed-source and privacy-compliant open-source models","volume":"35","author":"Salam","journal-title":"Eur Radiol"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.242575","article-title":"Generative large language models trained for detecting errors in radiology reports","volume":"315","author":"Sun","journal-title":"Radiology"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","journal-title":"Sci Data"},{"key":"R13","doi-asserted-by":"crossref","unstructured":"Irvin J Rajpurkar P Ko M et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Presented at: Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI 2019); Jan 27 to Feb 1, 2019; Honolulu, Hawaii, USA. [doi: 10.1609\/aaai.v33i01.3301590]","DOI":"10.1609\/aaai.v33i01.3301590"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1093\/jamia\/ocv080","article-title":"Preparing a collection of radiology examinations for distribution and retrieval","volume":"23","author":"Demner-Fushman","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"R15","doi-asserted-by":"publisher","DOI":"10.21037\/atm.2018.08.11","article-title":"Detecting insertion, substitution, and deletion errors in radiology reports using neural sequence-to-sequence models","volume":"7","author":"Zech","journal-title":"Ann Transl Med"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Min D Kim K Lee JH Kim Y Park CM . RRED: a radiology report error detector based on deep learning framework. Presented at: Proceedings of the 4th Clinical Natural Language Processing Workshop; Jul 14, 2022:41-52; Seattle, Washington, USA. [doi: 10.18653\/v1\/2022.clinicalnlp-1.5]","DOI":"10.18653\/v1\/2022.clinicalnlp-1.5"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.210185","article-title":"Application of a domain-specific BERT for detection of speech recognition errors in radiology reports","volume":"4","author":"Chaudhari","journal-title":"Radiol Artif Intell"},{"key":"R18","unstructured":"Structured model outputs. OpenAI Developers. URL: https:\/\/developers.openai.com\/api\/docs\/guides\/structured-outputs [Accessed 20-08-2025]"},{"key":"R19","unstructured":"Models. OpenAI Developers. URL: https:\/\/developers.openai.com\/api\/docs\/models [Accessed 11-06-2025]"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1002\/hec.584","article-title":"The death of cost-minimization analysis?","volume":"10","author":"Briggs","journal-title":"Health Econ"},{"key":"R21","unstructured":"Krupp L Gei\u00dfler D Lukowicz P Karolus J . Towards sustainable web agents: a plea for transparency and dedicated metrics for energy consumption. arXiv. Preprint posted online on Feb 25, 2025. [doi: 10.48550\/ARXIV.2502.17903]"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Strubell E Ganesh A McCallum A . Energy and policy considerations for deep learning in NLP. Presented at: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; Jul 28 to Aug 2, 2019; Florence, Italy. [doi: 10.18653\/v1\/P19-1355]","DOI":"10.18653\/v1\/P19-1355"},{"key":"R23","unstructured":"Darves B . Physician specialty compensation trends: salaries on the rise, but increases mostly modest. NEJM CareerCenter Resources. 2025. URL: https:\/\/resources.nejmcareercenter.org\/article\/physician-specialty-compensation-trends-salaries-on-the-rise-but-increases-mostly-modest\/ [Accessed 09-05-2026]"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","article-title":"The use of confidence or fiducial limits illustrated in the case of the binomial","volume":"26","author":"Clopper","journal-title":"Biometrika"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1162\/rest.90.3.414","article-title":"Bootstrap-based improvements for inference with clustered errors","volume":"90","author":"Cameron","journal-title":"Rev Econ Stat"},{"issue":"8","key":"R26","doi-asserted-by":"publisher","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","journal-title":"JAMA Intern Med"},{"issue":"4","key":"R27","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooae133","article-title":"External validation of the Epic sepsis predictive model in 2 county emergency departments","volume":"7","author":"Ostermayer","journal-title":"JAMIA Open"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1002\/cpt.2624","article-title":"Contextualized drug-drug interaction management improves clinical utility compared with basic drug-drug interaction management in hospitalized patients","volume":"112","author":"Wasylewicz","journal-title":"Clin Pharmacol Ther"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","DOI":"10.1177\/14604582241263242","article-title":"Override rate of drug-drug interaction alerts in clinical decision support systems: a brief systematic review and meta-analysis","volume":"30","author":"Felisberto","journal-title":"Health Informatics J"},{"issue":"6","key":"R30","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.240624","article-title":"AI as a second reader can reduce radiologists\u2019 workload and increase accuracy in screening mammography","volume":"6","author":"Suri","journal-title":"Radiol Artif Intell"},{"key":"R31","unstructured":"MIMIC-III clinical database. PhysioNet. URL: https:\/\/physionet.org\/content\/mimiciii\/1.4\/ [Accessed 14-05-2026]"},{"key":"R32","unstructured":"CheXpert: a large chest radiograph dataset. Stanford ML Group. URL: https:\/\/stanfordmlgroup.github.io\/competitions\/chexpert\/ [Accessed 14-05-2026]"},{"key":"R33","unstructured":"Open-i: open access biomedical image search engine. National Library of Medicine. URL: https:\/\/openi.nlm.nih.gov\/ [Accessed 14-05-2026]"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e87368","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:14Z","timestamp":1785536714000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e87368"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,4]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/87368","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,4]]},"article-number":"v14i1e87368"}}