{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T16:28:02Z","timestamp":1792600082875,"version":"4.5.0"},"reference-count":41,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,2,24]],"date-time":"2022-02-24T00:00:00Z","timestamp":1645660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Genet."],"abstract":"<jats:p>Research on detecting Tuberculosis (TB) findings on chest radiographs (or Chest X-rays: CXR) using convolutional neural networks (CNNs) has demonstrated superior performance due to the emergence of publicly available, large-scale datasets with expert annotations and availability of scalable computational resources. However, these studies use only the frontal CXR projections, i.e., the posterior-anterior (PA), and the anterior-posterior (AP) views for analysis and decision-making. Lateral CXRs which are heretofore not studied help detect clinically suspected pulmonary TB, particularly in children. Further, Vision Transformers (ViTs) with built-in self-attention mechanisms have recently emerged as a viable alternative to the traditional CNNs. Although ViTs demonstrated notable performance in several medical image analysis tasks, potential limitations exist in terms of performance and computational efficiency, between the CNN and ViT models, necessitating a comprehensive analysis to select appropriate models for the problem under study. This study aims to detect TB-consistent findings in lateral CXRs by constructing an ensemble of the CNN and ViT models. Several models are trained on lateral CXR data extracted from two large public collections to transfer modality-specific knowledge and fine-tune them for detecting findings consistent with TB. We observed that the weighted averaging ensemble of the predictions of CNN and ViT models using the optimal weights computed with the Sequential Least-Squares Quadratic Programming method delivered significantly superior performance (MCC: 0.8136, 95% confidence intervals (CI): 0.7394, 0.8878, <jats:italic>p<\/jats:italic> &amp;lt; 0.05) compared to the individual models and other ensembles. We also interpreted the decisions of CNN and ViT models using class-selective relevance maps and attention maps, respectively, and combined them to highlight the discriminative image regions contributing to the final output. We observed that (i) the model accuracy is not related to disease region of interest (ROI) localization and (ii) the bitwise-AND of the heatmaps of the top-2-performing models delivered significantly superior ROI localization performance in terms of mean average precision [mAP@(0.1 0.6) = 0.1820, 95% CI: 0.0771,0.2869, <jats:italic>p<\/jats:italic> &amp;lt; 0.05], compared to other individual models and ensembles. The code is available at <jats:ext-link>https:\/\/github.com\/sivaramakrishnan-rajaraman\/Ensemble-of-CNN-and-ViT-for-TB-detection-in-lateral-CXR<\/jats:ext-link>.<\/jats:p>","DOI":"10.3389\/fgene.2022.864724","type":"journal-article","created":{"date-parts":[[2022,2,24]],"date-time":"2022-02-24T07:24:31Z","timestamp":1645687471000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["Detecting Tuberculosis-Consistent Findings in Lateral Chest X-Rays Using an Ensemble of CNNs and Vision Transformers"],"prefix":"10.3389","volume":"13","author":[{"given":"Sivaramakrishnan","family":"Rajaraman","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghada","family":"Zamzmi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Les R.","family":"Folio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sameer","family":"Antani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,2,24]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"d2304","DOI":"10.1136\/bmj.d2304","article-title":"How to Obtain the P Value from a Confidence Interval","volume":"343","author":"Altman","year":"2011","journal-title":"BMJ"},{"key":"B2","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1186\/s40537-021-00444-8","article-title":"Review of Deep Learning: Concepts, CNN Architectures, Challenges, Applications, Future Directions","volume":"8","author":"Alzubaidi","year":"2021","journal-title":"J. Big Data"},{"key":"B3","doi-asserted-by":"publisher","first-page":"101797","DOI":"10.1016\/j.media.2020.101797","article-title":"PadChest: A Large Chest X-ray Image Dataset with Multi-Label Annotated Reports","volume":"66","author":"Bustos","year":"2020","journal-title":"Med. Image Anal."},{"key":"B4","volume-title":"ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases","author":"d\u2019Ascoli","year":"2021"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/3-540-45014-9_1","article-title":"Ensemble Methods in Machine Learning","volume":"1857","author":"Dietterich","year":"2000","journal-title":"Mult. Classif Syst."},{"key":"B6","article-title":"Local-global Classifier Fusion for Screening Chest Radiographs","volume-title":"Proc. Of SPIE Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications","author":"Ding","year":"2017"},{"key":"B7","doi-asserted-by":"publisher","first-page":"e0202121","DOI":"10.1371\/journal.pone.0202121","article-title":"Poor Statistical Reporting, Inadequate Data Presentation and Spin Persist Despite Editorial Advice","volume":"13","author":"Diong","year":"2018","journal-title":"PLoS One"},{"key":"B8","doi-asserted-by":"publisher","first-page":"115519","DOI":"10.1016\/j.eswa.2021.115519","article-title":"Detection of Tuberculosis from Chest X-ray Images: Boosting the Performance with Vision Transformer and Transfer Learning","volume":"184","author":"Duong","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"B9","first-page":"2276","article-title":"The VIA Annotation Software for Images, Audio and Video","author":"Dutta","year":"2019"},{"key":"B10","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1258\/jrsm.98.7.31010.1177\/014107680509800705","article-title":"Lateral Chest X-ray for Physicians","volume":"98","author":"Gaber","year":"2005","journal-title":"J. R. Soc. Med."},{"key":"B11","first-page":"740","article-title":"Microsoft COCO","volume-title":"European Conference on Computer Vision","author":"Gtua","year":"2014"},{"key":"B12","first-page":"2","article-title":"An Ensemble Model for Breast Cancer Prediction Using Sequential Least Squares Programming Method (SLSQP)","author":"Gupta","year":"2018"},{"key":"B13","first-page":"770","article-title":"Deep Residual Learning for Image Recognition","author":"He","year":"2016"},{"key":"B14","doi-asserted-by":"publisher","first-page":"8","DOI":"10.3389\/fpubh.2020.00016","article-title":"Review of Evidence for Using Chest X-Rays for Active Tuberculosis Screening in Long-Term Care in Canada","volume":"8","author":"Herrera Diaz","year":"2020","journal-title":"Front. Public Health"},{"key":"B15","first-page":"650","article-title":"Fusion of Local and Global Detection Systems to Detect Tuberculosis in Chest Radiographs","volume-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics","author":"Hogeweg","year":"2010"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.243","article-title":"Densely Connected Convolutional Networks","author":"Huang","year":"2017"},{"key":"B17","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1609\/aaai.v33i01.3301590","article-title":"CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison","volume":"33","author":"Irvin","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intelligence"},{"key":"B18","volume-title":"Abnormality Detection and Localization in Chest X-Rays Using Deep Convolutional Neural Networks","author":"Islam","year":"2017"},{"key":"B19","doi-asserted-by":"publisher","first-page":"475","DOI":"10.3978\/j.issn.2223-4292.2014.11.20","article-title":"Two Public Chest X-ray Datasets for Computer-Aided Screening of Pulmonary Diseases","volume":"4","author":"Jaeger","year":"2014","journal-title":"Quant Imaging Med. Surg."},{"key":"B20","doi-asserted-by":"publisher","first-page":"38","DOI":"10.3390\/diagnostics9020038","article-title":"Visual Interpretation of Convolutional Neural Network Predictions in Classifying Medical Image Modalities","volume":"9","author":"Kim","year":"2019","journal-title":"Diagnostics"},{"key":"B21","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1148\/radiol.2017162326","article-title":"Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks","volume":"284","author":"Lakhani","year":"2017","journal-title":"Radiology"},{"key":"B22","doi-asserted-by":"publisher","first-page":"012175","DOI":"10.1088\/1742-6596\/2010\/1\/012175","article-title":"Automatic Diagnosis of COVID-19 Using a Tailored Transformer-like Network","volume":"2010","author":"Liu","year":"2021","journal-title":"J. Phys. Conf. Ser."},{"key":"B23","doi-asserted-by":"publisher","first-page":"102299","DOI":"10.1016\/j.media.2021.102299","article-title":"Multi-task Vision Transformer Using Low-Level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification","volume":"75","author":"Park","year":"2022","journal-title":"Med. Image Anal."},{"key":"B24","doi-asserted-by":"publisher","first-page":"6268","DOI":"10.1038\/s41598-019-42557-4","article-title":"Efficient Deep Network Architectures for Fast Chest X-Ray Tuberculosis Screening and Visualization","volume":"9","author":"Pasa","year":"2019","journal-title":"Sci. Rep."},{"key":"B25","doi-asserted-by":"publisher","first-page":"27318","DOI":"10.1109\/ACCESS.2020.2971257","article-title":"Modality-Specific Deep Learning Model Ensembles toward Improving TB Detection in Chest Radiographs","volume":"8","author":"Rajaraman","year":"2020","journal-title":"IEEE Access"},{"key":"B26","first-page":"718","article-title":"A Novel Stacked Generalization of Models for Improved TB Detection in Chest Radiographs","author":"Rajaraman","year":"2018"},{"key":"B27","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.3390\/app8101715","article-title":"Visualization and Interpretation of Convolutional Neural Network Predictions in Detecting Pneumonia in Pediatric Chest Radiographs","volume":"8","author":"Rajaraman","year":"2018","journal-title":"Appl. Sci."},{"key":"B28","doi-asserted-by":"crossref","DOI":"10.1109\/EMBC.2019.8856715","article-title":"Assessment of an Ensemble of Machine Learning Models toward Abnormality Detection in Chest Radiographs","author":"Rajaraman","year":"2019"},{"key":"B29","doi-asserted-by":"publisher","first-page":"115041","DOI":"10.1109\/ACCESS.2020.3003810","article-title":"Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-Rays","volume":"8","author":"Rajaraman","year":"2020","journal-title":"IEEE Access"},{"key":"B30","doi-asserted-by":"publisher","first-page":"e8693","DOI":"10.7717\/peerj.8693","article-title":"Detection and Visualization of Abnormality in Chest Radiographs Using Modality-specific Convolutional Neural Network Ensembles","volume":"8","author":"Rajaraman","year":"2020","journal-title":"PeerJ"},{"key":"B31","volume-title":"CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning","author":"Rajpurkar","year":"2017"},{"key":"B32","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep Learning in Neural Networks: An Overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural Networks"},{"key":"B33","first-page":"618","article-title":"Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization","author":"Selvaraju","year":"2017"},{"key":"B34","doi-asserted-by":"publisher","first-page":"11086","DOI":"10.3390\/ijerph182111086","article-title":"Covid-transformer: Interpretable Covid-19 Detection Using Vision Transformer for Healthcare","volume":"18","author":"Shome","year":"2021","journal-title":"Int. J. Environ. Res. Public Health"},{"key":"B35","article-title":"Very Deep Convolutional Networks for Large-Scale Image Recognition","author":"Simonyan","year":"2015"},{"key":"B36","doi-asserted-by":"crossref","DOI":"10.1117\/12.2293140","article-title":"Comparing Deep Learning Models for Population Screening Using Chest Radiography","volume-title":"SPIE Medical Imaging","author":"Sivaramakrishnan","year":"2018"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1136\/adc.2004.062315","article-title":"Diagnostic Accuracy of Chest Radiography in Detecting Mediastinal Lymphadenopathy in Suspected Pulmonary Tuberculosis","volume":"90","author":"Swingler","year":"2005","journal-title":"Arch. Dis. Child."},{"key":"B38","volume-title":"EfficientNetV2: Smaller Models and Faster Training","author":"Tan","year":"2021"},{"key":"B39","first-page":"1","article-title":"ChestX-ray8: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases","author":"Wang","year":"2017"},{"key":"B40","first-page":"1","volume-title":"Chest Radiography in Tuberculosis","year":"2016"},{"key":"B41","article-title":"The Ninth Annual Conference on Learning Representations (ICLR 2021)","author":"Zhai","year":"2021"}],"container-title":["Frontiers in Genetics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fgene.2022.864724\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,24]],"date-time":"2022-02-24T07:24:45Z","timestamp":1645687485000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fgene.2022.864724\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,24]]},"references-count":41,"alternative-id":["10.3389\/fgene.2022.864724"],"URL":"https:\/\/doi.org\/10.3389\/fgene.2022.864724","relation":{},"ISSN":["1664-8021"],"issn-type":[{"value":"1664-8021","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,24]]},"article-number":"864724"}}