{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T17:35:00Z","timestamp":1791999300729,"version":"4.4.0"},"reference-count":32,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T00:00:00Z","timestamp":1769126400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: Interpretation of chest X-rays (CXRs) requires accurate identification of lesion presence, diagnosis, location, size, and number to be considered complete. However, the effectiveness of large language models with vision capabilities (LLMs) in performing these tasks remains uncertain. This study aimed to evaluate the image-only interpretation performance of LLMs in the absence of clinical information. Methods: A total of 247 CXRs covering 13 diagnostic categories, including pulmonary edema, cardiomegaly, lobar pneumonia, and other conditions, were evaluated using Gemini 1.0, Gemini 1.5 Pro, GPT-4 Turbo, and GPT-4o. The text outputs generated by the LLMs were evaluated at two levels: (1) primary diagnosis accuracy across the 13 predefined diagnostic categories, and (2) identification of key imaging features described in the generated text. Primary diagnosis accuracy was assessed based on whether the model correctly identified the target diagnostic category and was classified as fully correct, partially correct, or incorrect according to predefined clinical criteria. Non-diagnostic imaging features, such as posteroanterior and anteroposterior (PA\/AP) views, side markers, foreign bodies, and devices, were recorded and analyzed separately rather than being incorporated into the primary diagnostic scoring. Results: When fully and partially correct responses were treated as successful detections, vLLMs showed higher sensitivity for large, bilateral, multiple lesions and prominent devices, including acute pulmonary edema, lobar pneumonia, multiple malignancies, massive pleural effusions, and pacemakers, all of which demonstrated statistically significant differences across categories in chi-square analyses. Feature descriptions varied among models, especially in PA\/AP views and side markers, though central lines were partially recognized. Across the entire dataset, Gemini 1.5 Pro achieved the highest overall detection rate, followed by Gemini 1.0, GPT-4o, and GPT-4 Turbo. Conclusions: Although LLMs were able to identify certain diagnoses and key imaging features, their limitations in detecting small lesions, recognizing laterality, reasoning through differential diagnoses, and using domain-specific expressions indicate that CXR interpretation without textual cues still requires further improvement.<\/jats:p>","DOI":"10.3390\/diagnostics16030376","type":"journal-article","created":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T17:52:38Z","timestamp":1769190758000},"page":"376","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Limitations in Chest X-Ray Interpretation by Vision-Capable Large Language Models, Gemini 1.0, Gemini 1.5 Pro, GPT-4 Turbo, and GPT-4o"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-4387-0360","authenticated-orcid":false,"given":"Chih-Hsiung","family":"Chen","sequence":"first","affiliation":[{"name":"Department of Critical Care Medicine, Mennonite Christian Hospital, Hualien 970472, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang-Wei","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Emergency, Mennonite Christian Hospital, Hualien 970472, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kuang-Yu","family":"Hsieh","sequence":"additional","affiliation":[{"name":"Department of Critical Care Medicine, Mennonite Christian Hospital, Hualien 970472, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kuo-En","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Critical Care Medicine, Mennonite Christian Hospital, Hualien 970472, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2047-3481","authenticated-orcid":false,"given":"Hsien-Yung","family":"Lai","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, Da-Chien Health Medical System, Miaoli 36052, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.1007\/s12065-022-00775-2","article-title":"Simplified U-Net as a deep learning intelligent medical assistive tool in glaucoma detection","volume":"17","author":"Panahi","year":"2024","journal-title":"Evol. Intel."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/s44163-024-00110-x","article-title":"A multiclass deep learning algorithm for healthy lung, Covid-19 and pneumonia disease detection from chest X-ray images","volume":"4","author":"Mohan","year":"2024","journal-title":"Discov. Artif. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Oliveira, M., Seringa, J., Pinto, F.J., Henriques, R., and Magalhaes, T. (2023). Machine learning prediction of mortality in Acute Myocardial Infarction. BMC Med. Inform. Decis. Mak., 23.","DOI":"10.1186\/s12911-023-02168-6"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5190","DOI":"10.1007\/s00167-023-07529-2","article-title":"Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information","volume":"31","author":"Kaarre","year":"2023","journal-title":"Knee Surg. Sports Traumatol. Arthrosc."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hang, C.N., and Ho, S.M. (2025, January 15\u201315). Personalized Vocabulary Learning through Images: Harnessing Multimodal Large Language Models for Early Childhood Education. Proceedings of the IEEE Integrated STEM Education Conference (ISEC), Princeton, NJ, USA.","DOI":"10.1109\/ISEC64801.2025.11147254"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"113114","DOI":"10.1016\/j.engappai.2025.113114","article-title":"Children\u2019s psychological recognition with a multimodal language model incorporating visual language features","volume":"163","author":"Chen","year":"2026","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_7","unstructured":"Open, AI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2023). GPT-4 Technical Report. arXiv, Available online: http:\/\/arxiv.org\/abs\/2303.08774."},{"key":"ref_8","unstructured":"Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., and Silver, D. (2023). on behalf of the Gemini Team Google; et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv."},{"key":"ref_9","first-page":"e55991","article-title":"Comparing the Performance of Popular Large Language Models on the National Board of Medical Examiners Sample Questions","volume":"16","author":"Abbas","year":"2024","journal-title":"Cureus"},{"key":"ref_10","first-page":"e67641","article-title":"Comparing Vision-Capable Models, GPT-4 and Gemini, with GPT-3.5 on Taiwan\u2019s Pulmonologist Exam","volume":"16","author":"Chen","year":"2024","journal-title":"Cureus"},{"key":"ref_11","unstructured":"National Institutes of Health Clinical Center (2025, December 21). Ronald, M.S. 10 Center Drive, Bethesda, MD 20892, September 2017. Available online: https:\/\/nihcc.app.box.com\/v\/ChestXray-NIHCC."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Fanni, S.C., Marcucci, A., Volpi, F., Valentino, S., Neri, E., and Romei, C. (2023). Artificial Intelligence-Based Software with CE Mark for Chest X-Ray Interpretation: Opportunities and Challenges Review. Diagnostics, 13.","DOI":"10.3390\/diagnostics13122020"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lee, R.W., Lee, K.H., Yun, J.S., Kim, M.S., and Choi, H.S. (2024). Comparative Analysis of M4CXR, an LLM-Based Chest X-Ray Report Generation Model, and ChatGPT in Radiological Interpretation. J. Clin. Med., 13.","DOI":"10.3390\/jcm13237057"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1562","DOI":"10.1002\/clc.24105","article-title":"The diagnostic value of chest X-ray scanning by the help of Artificial Intelligence in Heart Failure (ART-IN-HF)","volume":"46","author":"Celik","year":"2023","journal-title":"Clin. Cardiol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"051805","DOI":"10.1117\/1.JMI.10.5.051805","article-title":"Integration and evaluation of chest X-ray artificial intelligence in clinical practice","volume":"10","author":"Wong","year":"2023","journal-title":"J. Med. Imaging"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.7150\/ijms.76515","article-title":"Application of artificial intelligence in diagnosing COVID-19 disease symptoms on chest X-rays: A systematic review","volume":"19","author":"Kufel","year":"2022","journal-title":"Int. J. Med. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"McGrath, E.E., and Anderson, P.B. (2011). Diagnosis of pleural effusion: A systematic approach. Am. J. Crit. Care, 20.","DOI":"10.4037\/ajcc2011685"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/0002-9343(50)90341-X","article-title":"Intrapulmonary pleural effusion simulating elevation of the diaphragm","volume":"8","author":"Rothstein","year":"1950","journal-title":"Am. J. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","year":"2023","journal-title":"Commun. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1007\/s43681-023-00353-x","article-title":"The human role to guarantee an ethical AI in healthcare: A five-facts approach","volume":"5","author":"Iniesta","year":"2023","journal-title":"AI Ethics"},{"key":"ref_21","first-page":"e43313","article-title":"Artificial Hallucinations by Google Bard: Think Before You Leap","volume":"15","author":"Kumar","year":"2023","journal-title":"Cureus"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1007\/s11604-024-01552-0","article-title":"The impact of large language models on radiology: A guide for radiologists on the latest innovations in AI","volume":"42","author":"Nakaura","year":"2024","journal-title":"Jpn. J. Radiol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s41747-024-00457-x","article-title":"A retrieval-augmented chatbot based on GPT-4 provides appropriate differential diagnosis in gastrointestinal radiology: A proof of concept study","volume":"8","author":"Rau","year":"2024","journal-title":"Eur. Radiol. Exp."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1038\/s41746-024-01081-0","article-title":"Augmented non-hallucinating large language models as medical information curators","volume":"7","author":"Gilbert","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Viana Vargas, T., Pedrini, H., and Santanch\u00e8, A. (2024). LLM-Driven Chest X-Ray Report Generation With a Modular, Reduced-Size Architecture. Intelligent Systems. BRACIS, Springer.","DOI":"10.1007\/978-3-031-79032-4_14"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3566","DOI":"10.1007\/s00330-023-10384-x","article-title":"Evaluating the performance of Generative Pre-trained Transformer-4 (GPT-4) in standardizing radiology reports","volume":"34","author":"Hasani","year":"2024","journal-title":"Eur. Radiol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1177\/08465371221131200","article-title":"U\u201dAI\u201d Testing: User Interface and Usability Testing of a Chest X-ray AI Tool in a Simulated Real-World Workflow","volume":"74","author":"Cheung","year":"2023","journal-title":"Can. Assoc. Radiol. J."},{"key":"ref_28","unstructured":"Yao, Y., Wen, Z., Tong, Y., Tian, X., Li, X., Ma, X., Xu, D., and Gedeon, T. (2025). Simple Radiology VLLM Test-time Scaling with Thought Graph Traversal. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1038\/s41591-024-03302-1","article-title":"Collaboration between clinicians and vision\u2013language models in radiology report generation","volume":"31","author":"Tanno","year":"2025","journal-title":"Nat. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"506","DOI":"10.1007\/s00330-024-10902-5","article-title":"ChatGPT\u2019s diagnostic performance based on textual vs. visual information compared to radiologists\u2019 diagnostic performance in musculoskeletal radiology","volume":"35","author":"Horiuchi","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"25414","DOI":"10.1038\/s41598-024-76450-6","article-title":"The two-stage detection-after-segmentation model improves the accuracy of identifying subdiaphragmatic lesions","volume":"14","author":"Chen","year":"2024","journal-title":"Sci. Rep."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2025, December 21). ChestX-ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. Available online: https:\/\/openaccess.thecvf.com\/content_cvpr_2017\/papers\/Wang_ChestX-ray8_Hospital-Scale_Chest_CVPR_2017_paper.pdf.","DOI":"10.1109\/CVPR.2017.369"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/3\/376\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T18:02:30Z","timestamp":1769191350000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/3\/376"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,23]]},"references-count":32,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,2]]}},"alternative-id":["diagnostics16030376"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16030376","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,23]]}}}