{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:02:06Z","timestamp":1792789326708,"version":"4.5.0"},"reference-count":28,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T00:00:00Z","timestamp":1763164800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NeuralMed"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background: Chest radiography (CXR) is the most frequently performed radiological exam worldwide, but reporting backlogs, caused by a shortage of radiologists, remain a critical challenge in emergency care. Artificial intelligence (AI) triage systems can help alleviate this challenge by differentiating normal from abnormal studies and prioritizing urgent cases for review. This study aimed to externally validate TRIA, a commercial AI-powered CXR triage algorithm (NeuralMed, S\u00e3o Paulo, Brazil). Methods: TRIA employs a two-stage deep learning approach, comprising an image segmentation module that isolates the thoracic region, followed by a classification model trained to recognize common cardiopulmonary pathologies. We trained the system on 275,399 CXRs from multiple public and private datasets. We performed external validation retrospectively on 1045 CXRs (568 normal and 477 abnormal) from a teaching university hospital that was not used for training. We established ground truth using a large language model (LLM) to extract findings from original radiologist reports. An independent radiologist review of a 300-report subset confirmed the reliability of this method, achieving an accuracy of 0.98 (95% CI 0.978\u20130.988). We compared four ensemble decision strategies for abnormality detection. Performance metrics included sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve (AUROC) with 95% CI. Results: The general abnormality classifier achieved strong performance (AUROC 0.911). Individual pathology models for cardiomegaly, pneumothorax, and effusion showed excellent results (AUROC of 0.968, 0.955, and 0.935, respectively). The weighted ensemble demonstrated the best balance, with an accuracy of 0.854 (95% CI, 0.831\u20130.874), a sensitivity of 0.845 (0.810\u20130.875), a specificity of 0.861 (0.830\u20130.887), and an AUROC of 0.927 (0.911\u20130.940). Sensitivity-prioritized methods achieving sensitivity &gt;0.92 produced lower specificity (&lt;0.69). False negatives were mainly subtle or equivocal cases, although many were still flagged as abnormal by the general classifier. Conclusions: TRIA achieved robust and balanced accuracy in distinguishing normal from abnormal CXRs. Integrating this system into clinical workflows has the potential to reduce reporting delays, prioritize urgent cases, and improve patient safety. These findings support its clinical utility and warrant prospective multicenter validation.<\/jats:p>","DOI":"10.3390\/diagnostics15222899","type":"journal-article","created":{"date-parts":[[2025,11,17]],"date-time":"2025-11-17T15:38:09Z","timestamp":1763393889000},"page":"2899","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["External Validation of an Artificial Intelligence Triaging System for Chest X-Rays: A Retrospective Independent Clinical Study"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8742-2801","authenticated-orcid":false,"given":"Andr\u00e9 Coutinho","family":"Castilla","sequence":"first","affiliation":[{"name":"NeuralMed A2 Tecnologia Ltda., Rua Pe Jo\u00e3o Manoel 1212, Cj 41, S\u00e3o Paulo 01411-010, SP, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iago de Paiva","family":"D\u2019Amorim","sequence":"additional","affiliation":[{"name":"Department of Radiology and Oncology, Hospital das Cl\u00ednicas, Faculdade de Medicina, Universidade de S\u00e3o Paulo (HCFMUSP), Av. Dr. En\u00e9as de Carvalho Aguiar 255, S\u00e3o Paulo 05403-000, SP, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0882-5050","authenticated-orcid":false,"given":"Maria Fernanda Barbosa","family":"Wanderley","sequence":"additional","affiliation":[{"name":"NeuralMed A2 Tecnologia Ltda., Rua Pe Jo\u00e3o Manoel 1212, Cj 41, S\u00e3o Paulo 01411-010, SP, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3537-9730","authenticated-orcid":false,"given":"Mateus Arag\u00e3o","family":"Esmeraldo","sequence":"additional","affiliation":[{"name":"Department of Radiology, Stanford University School of Medicine, Palo Alto, CA 94305, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andr\u00e9 Ricca","family":"Yoshida","sequence":"additional","affiliation":[{"name":"NeuralMed A2 Tecnologia Ltda., Rua Pe Jo\u00e3o Manoel 1212, Cj 41, S\u00e3o Paulo 01411-010, SP, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anthony Moreno","family":"Eigier","sequence":"additional","affiliation":[{"name":"NeuralMed A2 Tecnologia Ltda., Rua Pe Jo\u00e3o Manoel 1212, Cj 41, S\u00e3o Paulo 01411-010, SP, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9424-9776","authenticated-orcid":false,"given":"M\u00e1rcio","family":"Valente Yamada Sawamura","sequence":"additional","affiliation":[{"name":"Department of Radiology and Oncology, Hospital das Cl\u00ednicas, Faculdade de Medicina, Universidade de S\u00e3o Paulo (HCFMUSP), Av. Dr. En\u00e9as de Carvalho Aguiar 255, S\u00e3o Paulo 05403-000, SP, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,11,15]]},"reference":[{"key":"ref_1","first-page":"2400","article-title":"Use of Diagnostic Imaging Studies and Associated Radiation Exposure for Patients Enrolled in Large Integrated Health Care Systems, 1996\u20132010","volume":"307","author":"Miglioretti","year":"2012","journal-title":"JAMA"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1148\/radiol.2018180921","article-title":"Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks","volume":"291","author":"Annarumma","year":"2019","journal-title":"Radiology"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Anderson, P.G., Tarder-Stoll, H., Alpaslan, M., Keathley, N., Levin, D.L., Venkatesh, S., Bartel, E., Sicular, S., Howell, S., and Lindsey, R.V. (2024). Deep learning improves physician accuracy in the comprehensive detection of abnormalities on chest X-rays. Sci. Rep., 14.","DOI":"10.1038\/s41598-024-76608-2"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e258052","DOI":"10.1001\/jamanetworkopen.2025.8052","article-title":"Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use","volume":"8","author":"Windecker","year":"2025","journal-title":"JAMA Netw. Open"},{"key":"ref_6","unstructured":"Ag\u00eancia Nacional de Vigil\u00e2ncia Sanit\u00e1ria (2025, October 10). Consulta de Registro de Software como Dispositivo M\u00e9dico-A2 Tecnologia Ltda, Available online: https:\/\/consultas.anvisa.gov.br\/#\/saude\/25351120732202361\/?nomeProduto=tria."},{"key":"ref_7","unstructured":"Ag\u00eancia Nacional de Vigil\u00e2ncia Sanit\u00e1ria (2025, November 08). Disp\u00f5e Sobre as Boas Pr\u00e1ticas de Fabrica\u00e7\u00e3o de Produtos M\u00e9dicos e Produtos para Diagn\u00f3stico de Uso In Vitro e d\u00e1 Outras Provid\u00eancias, Available online: https:\/\/in.gov.br\/pt\/web\/dou\/-\/resolucao-de-diretoria-colegiada-rdc-n-657-de-24-de-marco-de-2022-389603457."},{"key":"ref_8","unstructured":"Brasil (2025, October 05). Lei Geral de Prote\u00e7\u00e3o de Dados Pessoais (Lei n\u00ba 13.709), Available online: http:\/\/www.planalto.gov.br\/ccivil_03\/_ato2015-2018\/2018\/lei\/l13709.htm."},{"key":"ref_9","first-page":"475","article-title":"Two public chest X-ray datasets for computer-aided screening of pulmonary diseases","volume":"4","author":"Jaeger","year":"2014","journal-title":"Quant. Imaging Med. Surg."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1093\/jamia\/ocv080","article-title":"Preparing a collection of radiology examinations for distribution and retrieval","volume":"23","author":"Kohli","year":"2016","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"71","DOI":"10.2214\/ajr.174.1.1740071","article-title":"Development of a digital image database for chest radiographs with and without a lung nodule: Receiver operating characteristic analysis of radiologists\u2019 detection of pulmonary nodules","volume":"174","author":"Shiraishi","year":"2000","journal-title":"Am. J. Roentgenol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2017, January 21\u201326). ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.369"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","article-title":"PadChest: A large chest X-ray image dataset with multi-label annotated reports","volume":"66","author":"Bustos","year":"2020","journal-title":"Med. Image Anal."},{"key":"ref_14","unstructured":"Gemini Team (2025, October 05). Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Available online: https:\/\/storage.googleapis.com\/deepmind-media\/gemini\/gemini_v2_5_report.pdf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","article-title":"Index for rating diagnostic tests","volume":"3","author":"Youden","year":"1950","journal-title":"Cancer"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Blake, S.R., Das, N., Tadepalli, M., Reddy, B., Singh, A., Agrawal, R., Chattoraj, S., Shah, D., and Putha, P. (2023). Using Artificial Intelligence to Stratify Normal versus Abnormal Chest X-rays: External Validation of a Deep Learning Algorithm at East Kent Hospitals University NHS Foundation Trust. Diagnostics, 13.","DOI":"10.3390\/diagnostics13223408"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Vasilev, Y., Vladzymyrskyy, A., Omelyanskaya, O., Blokhin, I., Kirpichev, Y., and Arzamasov, K. (2023). AI-Based CXR First Reading: Current Limitations to Ensure Practical Value. Diagnostics, 13.","DOI":"10.3390\/diagnostics13081430"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Harris, M., Qi, A., Jeagal, L., Torabi, N., Menzies, D., Korobitsyn, A., Pai, M., Nathavitharana, R.R., and Ahmad Khan, F. (2019). A systematic review of the diagnostic accuracy of artificial intelligence-based computer programs to analyze chest X-rays for pulmonary tuberculosis. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0221339"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Pepe, M.S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction, Oxford University Press.","DOI":"10.1093\/oso\/9780198509844.001.0001"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"e231236","DOI":"10.1148\/radiol.231236","article-title":"Commercially Available Chest Radiograph AI Tools for Detecting Airspace Disease, Pneumothorax, and Pleural Effusion","volume":"308","author":"Laustrup","year":"2023","journal-title":"Radiology"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Niehoff, J.H., Kalaitzidis, J., Kroeger, J.R., Schoenbeck, D., Borggrefe, J., and Michael, A.E. (2023). Evaluation of the clinical performance of an AI-based application for the automated analysis of chest X-rays. Sci. Rep., 13.","DOI":"10.1038\/s41598-023-30521-2"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e220510","DOI":"10.1148\/radiol.220510","article-title":"Artificial Intelligence for Clinical Interpretation of Bedside Chest Radiographs","volume":"307","author":"Khader","year":"2023","journal-title":"Radiology"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"de Camargo, T.F.O., Ribeiro, G.A.S., da Silva, M.C.B., da Silva, L.O., Torres, P.P.T.E.S., da Silva, D.D.S.R., de Santos, M.O.N., Filho, W.S., Rosa, M.E.E., and Novaes, M.D.A. (2025). Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs. Front. Artif. Intell., 8.","DOI":"10.3389\/frai.2025.1512910"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"e230981","DOI":"10.1148\/radiol.230981","article-title":"Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction","volume":"310","author":"Schalekamp","year":"2024","journal-title":"Radiology"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Irvin, J., Ball, R.L., Zhu, K., Yang, B., Mehta, H., Duan, T., Dsing, D., Bagul, A., and Langlotz, C.P. (2018). Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med., 15.","DOI":"10.1371\/journal.pmed.1002686"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Catalina, Q.M., Vidal-Alaball, J., Fuster-Casanovas, A., Escal\u00e9-Besa, A., Ruiz Comellas, A., and Sol\u00e9-Casals, J. (2024). Real-world testing of an artificial intelligence algorithm for the analysis of chest X-rays in primary care settings. Sci. Rep., 14.","DOI":"10.1038\/s41598-024-55792-1"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Qin, Z.Z., Sander, M.S., Rai, B., Titahong, C.N., Sudrungrot, S., Laah, S.N., Adhikari, L.M., Carter, E.J., Puri, L., and Codlin, A.J. (2019). Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy of three deep learning systems. Sci. Rep., 9.","DOI":"10.1038\/s41598-019-51503-3"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"5288","DOI":"10.21037\/qims-24-160","article-title":"Independent evaluation of the accuracy of 5 artificial intelligence software for detecting lung nodules on chest X-rays","volume":"14","author":"Arzamasov","year":"2024","journal-title":"Quant. Imaging Med. Surg."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/22\/2899\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,17]],"date-time":"2025-11-17T15:52:04Z","timestamp":1763394724000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/22\/2899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,15]]},"references-count":28,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["diagnostics15222899"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics15222899","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,15]]}}}