{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:36:35Z","timestamp":1787330195522,"version":"build-2736575974"},"reference-count":25,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) show promise in automatically detecting errors in radiology reports, but their performance remains insufficiently validated in large-scale, real-world clinical datasets.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to systematically evaluate the performance of LLMs in detecting and correcting errors in Chinese radiology reports derived from authentic clinical data.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A large-scale dataset of 4480 Chinese radiology reports with modification records containing real clinical practice-generated errors was retrospectively collected between January 2023 and June 2024 at a single institution. After exclusions, 1363 reports containing 1551 errors were included. The dataset covers various anatomical parts of the body from different imaging modalities and was randomly divided into a test set (n=1263) and an internal validation set (n=100). Additionally, 100 error-free reports were added to the internal validation set. An additional 200 English-language reports from the Medical Information Mart for Intensive Care (MIMIC-III) were used for external validation. Eight human readers and 8 widely adopted LLMs, enhanced by prompt engineering, were tasked with error detection. Overall and subgroup detection performance and reading time were evaluated. Correction suggestions from the 2 best-performing LLMs were reviewed by a senior radiologist.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n On the test set, DeepSeek-R1 achieved the highest overall detection rate at 89% (95% CI 87%-90%), significantly better than the other 7 models (\n P<\/jats:italic>\n =.001-.007). On the internal validation set, DeepSeek-R1 and Claude-3.5-Sonnet achieved detection rates of 83% (100\/120; 95% CI 76%-89%) and 80% (96\/120; 95% CI 72%-86%), respectively. DeepSeek-R1 showed performance comparable to radiologists (83%, 95% CI 76%-89% vs 80%, 95% CI 72%-86% for junior radiologists and 78%, 95% CI 70%-85% for senior radiologists;\n P<\/jats:italic>\n =.39 and\n P<\/jats:italic>\n =.19, respectively) and significantly better performance than that of nonradiologists and nonphysicians (83%, 95% CI 76%-89% vs 66%, 95% CI 57%-74% and 38%, 95% CI 30%-47%;\n P<\/jats:italic>\n <.001, respectively). DeepSeek-R1 showed a false-positive rate comparable to radiologists (DeepSeek-R1 vs senior radiologists and junior radiologists, 3% vs 0% and 1%;\n P<\/jats:italic>\n =.25 and\n P<\/jats:italic>\n =.61, respectively) and a significantly lower rate than nonradiologists and nonphysicians (3% vs 13% and 17%;\n P<\/jats:italic>\n =.02 and\n P<\/jats:italic>\n =.002, respectively). On the external validation set, DeepSeek-R1 and Claude-3.5-Sonnet achieved detection rates of 94% (95% CI 89%-97%) and 93% (95% CI 88%-97%), respectively. The correction accuracy of DeepSeek-R1 and Claude-3.5-Sonnet was 95% and 91%, respectively.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Enhanced LLMs, particularly DeepSeek-R1, demonstrated robust performance in error detection and correction within real-world Chinese radiology reports, supporting their clinical use for automated quality assurance and integration into workflows to improve reporting accuracy and efficiency.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/94689","type":"journal-article","created":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T15:51:55Z","timestamp":1787327515000},"page":"e94689-e94689","source":"Crossref","is-referenced-by-count":0,"title":["Error Detection and Correction in Chinese Radiology Reports Using Large Language Models: Real-World Clinical Validation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5404-2308","authenticated-orcid":false,"given":"Jiafeng","family":"Zhou","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7189-4343","authenticated-orcid":false,"given":"Yuxin","family":"Wei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9887-9997","authenticated-orcid":false,"given":"Qian","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9443-3760","authenticated-orcid":false,"given":"Yongchun","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0750-0774","authenticated-orcid":false,"given":"Eugene","family":"Edzeafene-Mensah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2080-2417","authenticated-orcid":false,"given":"Yunjun","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3683-6657","authenticated-orcid":false,"given":"Zhifang","family":"Pan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"376","DOI":"10.2214\/AJR.17.18721","article-title":"Radiology reporting: current practices and an introduction to patient-centered opportunities for improvement","volume":"210","author":"Mityul","journal-title":"AJR Am J Roentgenol"},{"issue":"2","key":"R2","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.251259","article-title":"Will generative large language models become radiologists\u2019 invaluable allies?","volume":"315","author":"Marrocchio","journal-title":"Radiology"},{"issue":"4","key":"R3","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1007\/s10278-015-9781-9","article-title":"Improving radiology report quality by rapidly notifying radiologist of report errors","volume":"28","author":"Minn","journal-title":"J Digit Imaging"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.242575","article-title":"Generative large language models trained for detecting errors in radiology reports","volume":"315","author":"Sun","journal-title":"Radiology"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"374","DOI":"10.1148\/radiol.2020201558","article-title":"Radiologists make more errors interpreting off-hours body CT studies during overnight assignments as compared with daytime assignments","volume":"297","author":"Patel","journal-title":"Radiology"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1148\/radiol.212631","article-title":"Mandating limits on workload, duty, and speed in radiology","volume":"304","author":"Alexander","journal-title":"Radiology"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"2257","DOI":"10.1109\/JBHI.2024.3514659","article-title":"A joint LLM-KG system for disease Q&A","volume":"29","author":"Sukhwal","journal-title":"IEEE J Biomed Health Inform"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232756","article-title":"Chatbots and large language models in radiology: a practical primer for clinical and research applications","volume":"310","author":"Bhayana","journal-title":"Radiology"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","DOI":"10.1148\/rg.240073","article-title":"Optimizing large language models in radiology and mitigating pitfalls: prompt engineering and fine-tuning","volume":"45","author":"Kim","journal-title":"Radiographics"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2025.109078","article-title":"Experience-guided multi-agent interpretable framework for radiology report summarization","volume":"273","author":"Li","journal-title":"Comput Methods Programs Biomed"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232714","article-title":"Potential of GPT-4 for detecting errors in radiology reports: implications for reporting accuracy","volume":"311","author":"Gertz","journal-title":"Radiology"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.230205","article-title":"Generative large language models for detection of speech recognition errors in radiology reports","volume":"6","author":"Schmidt","journal-title":"Radiol Artif Intell"},{"issue":"8","key":"R13","doi-asserted-by":"publisher","first-page":"4549","DOI":"10.1007\/s00330-025-11438-y","article-title":"Large language models for error detection in radiology reports: a comparative analysis between closed-source and privacy-compliant open-source models","volume":"35","author":"Salam","journal-title":"Eur Radiol"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232133","article-title":"BI-RADS category assignments by GPT-3.5, GPT-4, and Google Bard: a multilanguage study","volume":"311","author":"Cozzi","journal-title":"Radiology"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Lai V Ngo N Pouran Ben Veyseh A et al. ChatGPT beyond English: towards a comprehensive evaluation of large language models in multilingual learning. In: Bouamor H Pino J Bali K , editors. Findings of the Association for Computational Linguistics. 2023:13171-13189. [doi: 10.18653\/v1\/2023.findings-emnlp.878]","DOI":"10.18653\/v1\/2023.findings-emnlp.878"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Liu S Yang T Yue T Zhang F Wang D . PLOME: pre-training with misspelled knowledge for Chinese spelling correction. In: Zong C Xia F Li W Navigli R , editors. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021:2991-3000. [doi: 10.18653\/v1\/2021.acl-long.233]","DOI":"10.18653\/v1\/2021.acl-long.233"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1017\/ash.2025.10059","article-title":"Evaluating chain-of-thought prompting in a GPT chatbot for BCID2 interpretation and stewardship: how does AI compare to human experts?","volume":"5","author":"Tassone","journal-title":"Antimicrob Steward Healthc Epidemiol"},{"key":"R18","volume":"2024","author":"Shi","journal-title":"AMIA Annu Symp Proc"},{"issue":"28","key":"R19","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2502599122","article-title":"Asymptotic theory of in-context learning by linear attention","volume":"122","author":"Lu","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01468-7","article-title":"The use of large language models in detecting Chinese ultrasound report errors","volume":"8","author":"Yan","journal-title":"NPJ Digit Med"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-025-00275-0","article-title":"Scientists flock to DeepSeek: how they\u2019re using the blockbuster AI model","author":"Gibney","journal-title":"Nature"},{"issue":"3","key":"R22","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1111\/1754-9485.13621","article-title":"Current applications and future potential of ChatGPT in radiology: a systematic review","volume":"68","author":"Temperley","journal-title":"J Med Imaging Radiat Oncol"},{"issue":"4","key":"R23","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooae114","article-title":"Large language models and synthetic health data: progress and prospects","volume":"7","author":"Smolyak","journal-title":"JAMIA Open"},{"issue":"8","key":"R24","doi-asserted-by":"publisher","first-page":"2546","DOI":"10.1038\/s41591-025-03727-2","article-title":"Benchmark evaluation of DeepSeek large language models in clinical decision-making","volume":"31","author":"Sandmann","journal-title":"Nat Med"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240701","article-title":"Large-scale validation of the feasibility of GPT-4 as a proofreading tool for head CT reports","volume":"314","author":"Kim","journal-title":"Radiology"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T15:52:02Z","timestamp":1787327522000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e94689"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":25,"URL":"https:\/\/doi.org\/10.2196\/94689","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"article-number":"v28i2e94689"}}