{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T19:47:24Z","timestamp":1791229644899,"version":"4.1.0"},"reference-count":28,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T00:00:00Z","timestamp":1732492800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Despite significant strides in big data technology, extracting information from unstructured clinical data remains a formidable challenge. This study investigated the utility of large language models (LLMs) for extracting clinical data from unstructured radiological reports without additional training. In this retrospective study, 1800 radiologic reports, 600 from each of the three university hospitals, were collected, with seven pulmonary outcomes defined. Three pulmonology-trained specialists discerned the presence or absence of diseases. Data extraction from the reports was executed using Google Gemini Pro 1.0, OpenAI\u2019s GPT-3.5, and GPT-4. The gold standard was predicated on agreement between at least two pulmonologists. This study evaluated the performance of the three LLMs in diagnosing seven pulmonary diseases (active tuberculosis, emphysema, interstitial lung disease, lung cancer, pleural effusion, pneumonia, and pulmonary edema) utilizing chest radiography and computed tomography scans. All models exhibited high accuracy (0.85\u20131.00) for most conditions. GPT-4 consistently outperformed its counterparts, demonstrating a sensitivity of 0.71\u20131.00; specificity of 0.89\u20131.00; and accuracy of 0.89 and 0.99 across both modalities, thus underscoring its superior capability in interpreting radiological reports. Notably, the accuracy of pleural effusion and emphysema on chest radiographs and pulmonary edema on chest computed tomography scans reached 0.99. The proficiency of LLMs, particularly GPT-4, in accurately classifying unstructured radiological data hints at their potential as alternatives to the traditional manual chart reviews conducted by clinicians.<\/jats:p>","DOI":"10.1371\/journal.pone.0314136","type":"journal-article","created":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T18:31:09Z","timestamp":1732559469000},"page":"e0314136","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":16,"title":["Extraction of clinical data on major pulmonary diseases from unstructured radiologic reports using a large language model"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5523-5144","authenticated-orcid":true,"given":"Hyung Jun","family":"Park","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jin-Young","family":"Huh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7480-2634","authenticated-orcid":true,"given":"Ganghee","family":"Chae","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3538-1993","authenticated-orcid":true,"given":"Myeong Geun","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,11,25]]},"reference":[{"key":"pone.0314136.ref001","first-page":"1","article-title":"Big data in healthcare: management, analysis and future prospects","volume":"6","author":"S Dash","year":"2019","journal-title":"Jobd"},{"key":"pone.0314136.ref002","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"TH Kung","year":"2023","journal-title":"PLOS Digit Health"},{"key":"pone.0314136.ref003","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1186\/s44247-023-00058-5","article-title":"Evaluating GPT-4-based ChatGPT\u2019s clinical potential on the NEJM quiz","volume":"2","author":"D Ueda","year":"2024","journal-title":"BMC Digital Health"},{"key":"pone.0314136.ref004","first-page":"1018","article-title":"Beyond ChatGPT: What does GPT-4 add to healthcare? The dawn of a new era","volume":"30","author":"S W\u00f3jcik","year":"2023","journal-title":"Cardiol J"},{"key":"pone.0314136.ref005","article-title":"Evaluating the performance of Generative Pre-trained Transformer-4 (GPT-4) in standardizing radiology reports","author":"AM Hasani","year":"2023","journal-title":"Eur Radiol"},{"key":"pone.0314136.ref006","doi-asserted-by":"crossref","first-page":"101035","DOI":"10.1016\/j.jocmr.2024.101035","article-title":"Generative Pre-trained Transformer 4 makes cardiovascular magnetic resonance reports easy to understand","volume":"26","author":"B Salam","year":"2024","journal-title":"J Cardiovasc Magn Reson"},{"key":"pone.0314136.ref007","doi-asserted-by":"crossref","first-page":"e2325000","DOI":"10.1001\/jamanetworkopen.2023.25000","article-title":"Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis","volume":"6","author":"YF Shea","year":"2023","journal-title":"JAMA Netw Open"},{"key":"pone.0314136.ref008","doi-asserted-by":"crossref","first-page":"e48568","DOI":"10.2196\/48568","article-title":"Utility of ChatGPT in Clinical Practice","volume":"25","author":"J Liu","year":"2023","journal-title":"J Med Internet Res"},{"key":"pone.0314136.ref009","doi-asserted-by":"crossref","first-page":"20159","DOI":"10.1038\/s41598-023-47500-2","article-title":"A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports","volume":"13","author":"D Truhn","year":"2023","journal-title":"Sci Rep"},{"key":"pone.0314136.ref010","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"X Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"pone.0314136.ref011","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1038\/s41591-018-0335-9","article-title":"Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence","volume":"25","author":"H Liang","year":"2019","journal-title":"Nature Medicine"},{"key":"pone.0314136.ref012","doi-asserted-by":"crossref","first-page":"209","DOI":"10.3857\/roj.2023.00633","article-title":"Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer","volume":"41","author":"HS Choi","year":"2023","journal-title":"Radiat Oncol J"},{"key":"pone.0314136.ref013","doi-asserted-by":"crossref","first-page":"1180962","DOI":"10.3389\/fphar.2023.1180962","article-title":"Approach to machine learning for extraction of real-world data variables from electronic health records","volume":"14","author":"B Adamson","year":"2023","journal-title":"Front Pharmacol"},{"key":"pone.0314136.ref014","doi-asserted-by":"crossref","first-page":"e24803","DOI":"10.2196\/24803","article-title":"An Attention Model With Transfer Embeddings to Classify Pneumonia-Related Bilingual Imaging Reports: Algorithm Development and Validation","volume":"9","author":"H Park","year":"2021","journal-title":"JMIR Med Inform"},{"key":"pone.0314136.ref015","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1186\/s12911-022-01975-7","article-title":"Automated extraction of information of lung cancer staging from unstructured reports of PET-CT interpretation: natural language processing with deep-learning","volume":"22","author":"HJ Park","year":"2022","journal-title":"BMC Med Inform Decis Mak"},{"key":"pone.0314136.ref016","unstructured":"OpenAI. OpenAI API. [cited 2024 10 May][Internet]. https:\/\/openai.com\/index\/openai-api\/."},{"key":"pone.0314136.ref017","doi-asserted-by":"crossref","first-page":"e2305016120","DOI":"10.1073\/pnas.2305016120","article-title":"ChatGPT outperforms crowd workers for text-annotation tasks","volume":"120","author":"F Gilardi","year":"2023","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pone.0314136.ref018","doi-asserted-by":"crossref","first-page":"e231362","DOI":"10.1148\/radiol.231362","article-title":"Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer","volume":"308","author":"MA Fink","year":"2023","journal-title":"Radiology"},{"key":"pone.0314136.ref019","doi-asserted-by":"crossref","first-page":"e230725","DOI":"10.1148\/radiol.230725","article-title":"Leveraging GPT-4 for Post Hoc Transformation of Free-text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study","volume":"307","author":"LC Adams","year":"2023","journal-title":"Radiology"},{"key":"pone.0314136.ref020","doi-asserted-by":"crossref","first-page":"e18","DOI":"10.1164\/rccm.202202-0399ST","article-title":"Idiopathic Pulmonary Fibrosis (an Update) and Progressive Pulmonary Fibrosis in Adults: An Official ATS\/ERS\/JRS\/ALAT Clinical Practice Guideline","volume":"205","author":"G Raghu","year":"2022","journal-title":"Am J Respir Crit Care Med"},{"key":"pone.0314136.ref021","doi-asserted-by":"crossref","first-page":"e45","DOI":"10.1164\/rccm.201908-1581ST","article-title":"Diagnosis and Treatment of Adults with Community-acquired Pneumonia. An Official Clinical Practice Guideline of the American Thoracic Society and Infectious Diseases Society of America","volume":"200","author":"JP Metlay","year":"2019","journal-title":"Am J Respir Crit Care Med"},{"key":"pone.0314136.ref022","unstructured":"Network NCC. Lung cancer screening (Version 2.2024). https:\/\/www.nccn.org\/professionals\/physician_gls\/pdf\/lung_screening.pdfApril 26, 2024][Internet]. https:\/\/www.nccn.org\/professionals\/physician_gls\/pdf\/lung_screening.pdf."},{"key":"pone.0314136.ref023","doi-asserted-by":"crossref","first-page":"e231259","DOI":"10.1148\/radiol.231259","article-title":"Evaluating GPT4 on Impressions Generation in Radiology Reports","volume":"307","author":"Z Sun","year":"2023","journal-title":"Radiology"},{"key":"pone.0314136.ref024","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/S2213-2600(23)00461-7","article-title":"GOLD COPD report: 2024 update","volume":"12","author":"P. Venkatesan","year":"2024","journal-title":"Lancet Respir Med"},{"key":"pone.0314136.ref025","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1148\/rg.2017160032","article-title":"Pulmonary Tuberculosis: Role of Radiology in Diagnosis and Management","volume":"37","author":"AC Nachiappan","year":"2017","journal-title":"Radiographics"},{"key":"pone.0314136.ref026","doi-asserted-by":"crossref","first-page":"990","DOI":"10.1016\/j.jacr.2023.05.003","article-title":"Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot","volume":"20","author":"A Rao","year":"2023","journal-title":"J Am Coll Radiol"},{"key":"pone.0314136.ref027","doi-asserted-by":"crossref","first-page":"e232411","DOI":"10.1148\/radiol.232411","article-title":"Comparing GPT-3.5 and GPT-4 Accuracy and Drift in Radiology Diagnosis Please Cases","volume":"310","author":"D Li","year":"2024","journal-title":"Radiology"},{"key":"pone.0314136.ref028","article-title":"Artificial Intelligence in Multilingual Interpretation and Radiology Assessment for Clinical Language Evaluation (AI-MIRACLE)","author":"P Khanna","journal-title":"Journal of Personalized Medicine"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0314136","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T18:31:18Z","timestamp":1732559478000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0314136"}},"subtitle":[],"editor":[{"given":"Harpreet Singh","family":"Grewal","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,11,25]]},"references-count":28,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,11,25]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0314136","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,25]]}}}