{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T16:22:55Z","timestamp":1777998175002,"version":"3.51.4"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,2,6]],"date-time":"2025-02-06T00:00:00Z","timestamp":1738800000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>Chest X-ray (CXR) interpretation is critical in diagnosing various lung diseases. However, physicians, not specialists, are often the first ones to read them, frequently facing challenges in accurate interpretation. Artificial Intelligence (AI) algorithms could be of great help, but using real-world data is crucial to ensure their effectiveness in diverse healthcare settings. This study evaluates a deep learning algorithm designed for CXR interpretation, focusing on its utility for non-specialists in thoracic radiology physicians.<\/jats:p><\/jats:sec><jats:sec><jats:title>Purpose<\/jats:title><jats:p>To assess the performance of a Convolutional Neural Networks (CNNs)-based AI algorithm in interpreting CXRs and compare it with a team of physicians, including thoracic radiologists, who served as the gold-standard.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>A retrospective study from January 2021 to July 2023 evaluated an algorithm with three independent models for Lung Abnormality, Radiological Findings, and Tuberculosis. The algorithm's performance was measured using accuracy, sensitivity, and specificity. Two groups of physicians validated the model: one with varying specialties and experience levels in interpreting chest radiographs (Group A) and another of board-certified thoracic radiologists (Group B). The study also assessed the agreement between the two groups on the algorithm's heatmap and its influence on their decisions.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>In the internal validation, the Lung Abnormality and Tuberculosis models achieved an AUC of 0.94, while the Radiological Findings model yielded a mean AUC of 0.84. During the external validation, utilizing the ground truth generated by board-certified thoracic radiologists, the algorithm achieved better sensitivity in 6 out of 11 classes than physicians with varying experience levels. Furthermore, Group A physicians demonstrated higher agreement with the algorithm in identifying markings in specific lung regions than Group B (37.56% Group A vs. 21.75% Group B). Additionally, physicians declared that the algorithm did not influence their decisions in 93% of the cases.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>This retrospective clinical validation study assesses an AI algorithm's effectiveness in interpreting Chest X-rays (CXR). The results show the algorithm's performance is comparable to Group A physicians, using gold-standard analysis (Group B) as the reference. Notably, both Groups reported minimal influence of the algorithm on their decisions in most cases.<\/jats:p><\/jats:sec>","DOI":"10.3389\/frai.2025.1512910","type":"journal-article","created":{"date-parts":[[2025,2,6]],"date-time":"2025-02-06T07:10:31Z","timestamp":1738825831000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs"],"prefix":"10.3389","volume":"8","author":[{"given":"Thiago Fellipe Ortiz","family":"de Camargo","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guilherme Alberto Sousa","family":"Ribeiro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria Carolina Bueno","family":"da Silva","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luan Oliveira","family":"da Silva","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pedro Paulo Teixeira e Silva","family":"Torres","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Denise do Socorro da Silva","family":"Rodrigues","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mayler Olombrada Nunes","family":"de Santos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William Salibe","family":"Filho","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcela Emer Egypto","family":"Rosa","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Magdala de Araujo","family":"Novaes","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thiago Augusto","family":"Massarutto","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Osvaldo Landi","family":"Junior","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elaine","family":"Yanata","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcio Rodrigues da Cunha","family":"Reis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gilberto","family":"Szarf","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pedro Vieira Santana","family":"Netto","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joselisa Peres Queiroz","family":"de Paiva","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,2,6]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1002\/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5","article-title":"What do we mean by validating a prognostic model?","volume":"19","author":"Altman","year":"2000","journal-title":"Stat. Med"},{"key":"B2","volume-title":"Improving Diagnosis in Health Care","author":"Ball","year":"2015"},{"key":"B3","doi-asserted-by":"publisher","first-page":"8263","DOI":"10.1007\/s00330-023-09747-1","article-title":"Can incorrect artificial intelligence (ai) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography","volume":"33","author":"Bernstein","year":"2023","journal-title":"Eur. Radiol"},{"key":"B4","doi-asserted-by":"publisher","first-page":"3408","DOI":"10.3390\/diagnostics13223408","article-title":"Using artificial intelligence to stratify normal versus abnormal chest x-rays: external validation of a deep learning algorithm at east Kent Hospitals University NHS foundation trust","volume":"13","author":"Blake","year":"2023","journal-title":"Diagnostics"},{"key":"B5","doi-asserted-by":"publisher","first-page":"27","DOI":"10.3389\/fmed.2020.00027","article-title":"Artificial intelligence in medicine: today and tomorrow","volume":"7","author":"Briganti","year":"2020","journal-title":"Front. Med"},{"key":"B6","doi-asserted-by":"publisher","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","article-title":"Padchest: a large chest x-ray image dataset with multi-label annotated reports","volume":"66","author":"Bustos","year":"2020","journal-title":"Med. Image Anal"},{"key":"B7","doi-asserted-by":"publisher","first-page":"2113","DOI":"10.1148\/rg.2017170077","article-title":"Deep learning: a primer for radiologists. radiographics","volume":"37","author":"Chartrand","year":"2017","journal-title":"Radiographics"},{"key":"B8","doi-asserted-by":"publisher","first-page":"e44","DOI":"10.1016\/S2589-7500(23)00218-2","article-title":"Development and validation of open-source deep neural networks for comprehensive chest x-ray reading: a retrospective multicentre study","volume":"13","author":"Cid","year":"2024","journal-title":"Diagnostics"},{"key":"B9","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1093\/biomet\/26.4.404","article-title":"The use of confidence or fiducial limits illustrated in the case of the binomial","volume":"26","author":"Clopper","year":"1934","journal-title":"Biometrika"},{"key":"B10","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1038\/s41746-021-00549-7","article-title":"Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review","volume":"5","author":"de Hond","year":"2022","journal-title":"NPJ Dig. Med"},{"key":"B11","doi-asserted-by":"publisher","first-page":"118","DOI":"10.5152\/dir.2016.16187","article-title":"Missed lung cancer: when, where, and why?","volume":"23","author":"Del Ciello","year":"2017","journal-title":"Diag. Int. Radiol"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848","author":"Deng","year":"2009","journal-title":"2009 IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B13","doi-asserted-by":"publisher","first-page":"e657","DOI":"10.1016\/S2589-7500(22)00129-7","article-title":"Clinical validation of deep learning algorithms for radiotherapy targeting of non-small-cell lung cancer: an observational study","volume":"4","author":"Hosny","year":"2022","journal-title":"Lancet Digital Health"},{"key":"B14","doi-asserted-by":"publisher","first-page":"4797","DOI":"10.21037\/jtd-21-810","article-title":"A systematic review and meta-analysis of diagnostic performance and physicians' perceptions of artificial intelligence (AI)-assisted CT diagnostic technology for the classification of pulmonary nodules","volume":"13","author":"Huang","year":"2021","journal-title":"J. Thorac. Dis"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.3301590","author":"Irvin","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B16","doi-asserted-by":"publisher","first-page":"475","DOI":"10.3978\/j.issn.2223-4292.2014.11.20","article-title":"Two public chest x-ray datasets for computer-aided screening of pulmonary diseases","volume":"4","author":"Jaeger","year":"2014","journal-title":"Quant. Imaging Med. Surg"},{"key":"B17","doi-asserted-by":"publisher","first-page":"429","DOI":"10.3233\/IDA-2002-6504","article-title":"The class imbalance problem: a systematic study","volume":"6","author":"Japkowicz","year":"2002","journal-title":"Intell. Data Anal"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12916-019-1426-2","article-title":"Key challenges for delivering clinical impact with artificial intelligence","volume":"17","author":"Kelly","year":"2019","journal-title":"BMC Med"},{"key":"B19","doi-asserted-by":"publisher","first-page":"51747","DOI":"10.1109\/ACCESS.2021.3069937","article-title":"Intelligent pneumonia identification from chest x-rays: a systematic literature review","volume":"9","author":"Khan","year":"2021","journal-title":"IEEE Access"},{"key":"B20","doi-asserted-by":"publisher","first-page":"405","DOI":"10.3348\/kjr.2019.0025","article-title":"Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers","volume":"20","author":"Kim","year":"2019","journal-title":"Korean J. Radiol"},{"key":"B21","doi-asserted-by":"publisher","first-page":"754","DOI":"10.2214\/AJR.16.17224","article-title":"Implementing machine learning in radiology practice and research","volume":"208","author":"Kohli","year":"2017","journal-title":"Am. J. Roentgenol"},{"key":"B22","doi-asserted-by":"publisher","first-page":"2206","DOI":"10.3390\/diagnostics11122206","article-title":"The added effect of artificial intelligence on physicians' performance in detecting thoracic pathologies on CT and chest x-ray: a systematic review","volume":"11","author":"Li","year":"2021","journal-title":"Diagnostics"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1552","DOI":"10.17179\/excli2020-3095","article-title":"Artificial intelligence and its future potential in lung cancer screening","volume":"19","author":"Mathew","year":"2020","journal-title":"EXCLI J"},{"key":"B24","doi-asserted-by":"publisher","first-page":"2170","DOI":"10.1002\/sim.2677","article-title":"Confidence intervals for predictive values with an emphasis to case-control studies","volume":"26","author":"Mercaldo","year":"2007","journal-title":"Stat. Med"},{"key":"B25","doi-asserted-by":"publisher","first-page":"e221894","DOI":"10.1148\/radiol.221894","article-title":"AI improves nodule detection on chest radiographs in a health screening population: a randomized controlled trial","volume":"307","author":"Nam","year":"2023","journal-title":"Radiology"},{"key":"B26","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1148\/radiol.2018180237","article-title":"Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs","volume":"290","author":"Nam","year":"2019","journal-title":"Radiology"},{"key":"B27","doi-asserted-by":"publisher","first-page":"890759","DOI":"10.3389\/fdgth.2022.890759","article-title":"Deployment and validation of an ai system for detecting abnormal chest radiographs in clinical settings","volume":"4","author":"Nguyen","year":"2022","journal-title":"Front. Digital Health"},{"key":"B28","doi-asserted-by":"publisher","first-page":"e12422","DOI":"10.2196\/12422","article-title":"Physician confidence in artificial intelligence: an online mobile survey","volume":"21","author":"Oh","year":"2019","journal-title":"J. Med. Internet Res"},{"key":"B29","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1148\/radiol.2018182294","article-title":"Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance","volume":"290","author":"Park","year":"2019","journal-title":"Radiology"},{"key":"B30","doi-asserted-by":"publisher","first-page":"800","DOI":"10.1148\/radiol.2017171920","article-title":"Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction","volume":"286","author":"Park","year":"2018","journal-title":"Radiology"},{"key":"B31","doi-asserted-by":"publisher","first-page":"e152","DOI":"10.3346\/jkms.2018.33.e152","article-title":"Connecting technological innovation in artificial intelligence to real-world medical practice through rigorous clinical validation: what peer-reviewed medical journals could do","volume":"33","author":"Park","year":"2018","journal-title":"J. Korean Med. Sci"},{"key":"B32","author":"Paszke","year":"2019","journal-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems"},{"key":"B33","doi-asserted-by":"publisher","author":"Reis","year":"2022","DOI":"10.1038\/s41597-022-01608-8"},{"key":"B34","doi-asserted-by":"publisher","first-page":"3267","DOI":"10.1128\/JCM.01013-17","article-title":"The TB portals: an open-access, web-based platform for global drug-resistant-tuberculosis data sharing and analysis","volume":"55","author":"Rosenthal","year":"2017","journal-title":"J. Clin. Microbiol"},{"key":"B35","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1038\/s41746-019-0106-0","article-title":"Physician perspectives on integration of artificial intelligence into diagnostic pathology","volume":"2","author":"Sarwar","year":"2019","journal-title":"NPJ Dig. Med"},{"key":"B36","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.ejmp.2022.06.003","article-title":"Current challenges of implementing artificial intelligence in medical imaging","volume":"100","author":"Saw","year":"2022","journal-title":"Phys. Med"},{"key":"B37","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1148\/radiol.2019182465","article-title":"Deep convolutional neural network-based software improves radiologist detection of malignant lung nodules on chest radiographs","volume":"294","author":"Sim","year":"2020","journal-title":"Radiology"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-11149-6_2","author":"Tack","year":"2019","journal-title":"Diseases of the Chest, Breast, Heart and Vessels 2019\u20132022: Diagnostic and Interventional Imaging"},{"key":"B39","doi-asserted-by":"publisher","first-page":"e200190","DOI":"10.1148\/ryai.2021200190","article-title":"Deep learning systems for pneumothorax detection on chest radiographs: a multicenter external validation study","volume":"3","author":"Thian","year":"2021","journal-title":"Radiology"},{"key":"B40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12885-021-08847-9","article-title":"Artificial intelligence-supported lung cancer detection by multi-institutional readers with multi-vendor chest radiographs: a retrospective clinical validation study","volume":"21","author":"Ueda","year":"2021","journal-title":"BMC Cancer"},{"key":"B41","doi-asserted-by":"publisher","first-page":"924","DOI":"10.1038\/s41591-022-01772-9","article-title":"Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: decide-AI","volume":"28","author":"Vasey","year":"2022","journal-title":"Nat. Med"},{"key":"B42","doi-asserted-by":"publisher","first-page":"E1379","DOI":"10.1055\/a-1223-1926","article-title":"Physician sentiment toward artificial intelligence (ai) in colonoscopic practice: a survey of us gastroenterologists","volume":"8","author":"Wadhwa","year":"2020","journal-title":"Endos. Int. Open"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.369","author":"Wang","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B44","article-title":"Weakly supervised lesion localization with probabilistic-cam pooling","author":"Ye","year":"2020","journal-title":"arXiv preprint arXiv:2005.14480"},{"key":"B45","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1093\/nsr\/nwx106","article-title":"A brief introduction to weakly supervised learning","volume":"5","author":"Zhou","year":"2018","journal-title":"Natl. Sci. Rev"},{"key":"B46","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1200\/CCI.19.00131","article-title":"Open health imaging foundation viewer: an extensible open-source framework for building web-based imaging applications to support cancer research","volume":"4","author":"Ziegler","year":"2020","journal-title":"JCO Clin. Cancer Inform"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1512910\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,5]],"date-time":"2025-03-05T12:02:03Z","timestamp":1741176123000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1512910\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,6]]},"references-count":46,"alternative-id":["10.3389\/frai.2025.1512910"],"URL":"https:\/\/doi.org\/10.3389\/frai.2025.1512910","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,6]]},"article-number":"1512910"}}