{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T13:03:30Z","timestamp":1789131810678,"version":"build-2803163510"},"reference-count":36,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T00:00:00Z","timestamp":1640822400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/publicdomain\/zero\/1.0\/"}],"funder":[{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Medical images commonly exhibit multiple abnormalities. Predicting them requires multi-class classifiers whose training and desired reliable performance can be affected by a combination of factors, such as, dataset size, data source, distribution, and the loss function used to train deep neural networks. Currently, the cross-entropy loss remains the de-facto loss function for training deep learning classifiers. This loss function, however, asserts equal learning from all classes, leading to a bias toward the majority class. Although the choice of the loss function impacts model performance, to the best of our knowledge, we observed that no literature exists that performs a comprehensive analysis and selection of an appropriate loss function toward the classification task under study. In this work, we benchmark various state-of-the-art loss functions, critically analyze model performance, and propose improved loss functions for a multi-class classification task. We select a pediatric chest X-ray (CXR) dataset that includes images with no abnormality (normal), and those exhibiting manifestations consistent with bacterial and viral pneumonia. We construct prediction-level and model-level ensembles to improve classification performance. Our results show that compared to the individual models and the state-of-the-art literature, the weighted averaging of the predictions for top-3 and top-5 model-level ensembles delivered significantly superior classification performance (<jats:italic>p<\/jats:italic> &lt; 0.05) in terms of MCC (0.9068, 95% confidence interval (0.8839, 0.9297)) metric. Finally, we performed localization studies to interpret model behavior and confirm that the individual models and ensembles learned task-specific features and highlighted disease-specific regions of interest. The code is available at <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/sivaramakrishnan-rajaraman\/multiloss_ensemble_models\" xlink:type=\"simple\">https:\/\/github.com\/sivaramakrishnan-rajaraman\/multiloss_ensemble_models<\/jats:ext-link>.<\/jats:p>","DOI":"10.1371\/journal.pone.0261307","type":"journal-article","created":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T18:29:00Z","timestamp":1640888940000},"page":"e0261307","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":42,"title":["Novel loss functions for ensemble-based medical image classification"],"prefix":"10.1371","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0871-8634","authenticated-orcid":true,"given":"Sivaramakrishnan","family":"Rajaraman","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghada","family":"Zamzmi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0040-1387","authenticated-orcid":true,"given":"Sameer K.","family":"Antani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2021,12,30]]},"reference":[{"issue":"11","key":"pone.0261307.ref001","doi-asserted-by":"crossref","first-page":"e1002686","DOI":"10.1371\/journal.pmed.1002686","article-title":"Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists","volume":"15","author":"P Rajpurkar","year":"2018","journal-title":"PLoS Med."},{"issue":"11","key":"pone.0261307.ref002","doi-asserted-by":"crossref","first-page":"e0242301","DOI":"10.1371\/journal.pone.0242301","article-title":"Analyzing inter-reader variability affecting deep ensemble learning for COVID-19 detection in chest radiographs","volume":"15","author":"S Rajaraman","year":"2020","journal-title":"PLoS One"},{"key":"pone.0261307.ref003","first-page":"3462","article-title":"ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases","author":"X Wang","year":"2017","journal-title":"CVPR"},{"issue":"1","key":"pone.0261307.ref004","doi-asserted-by":"crossref","first-page":"e180041","DOI":"10.1148\/ryai.2019180041","article-title":"Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia","volume":"1","author":"G Shih","year":"2019","journal-title":"Radiol Artif Intell."},{"key":"pone.0261307.ref005","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1609\/aaai.v33i01.3301590","article-title":"CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison","author":"J Irvin","year":"2019","journal-title":"AAAI"},{"issue":"5","key":"pone.0261307.ref006","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1016\/j.cell.2018.02.010","article-title":"Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning","volume":"172","author":"DS Kermany","year":"2018","journal-title":"Cell"},{"issue":"2","key":"pone.0261307.ref007","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1148\/radiol.2017162326","article-title":"Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks.","volume":"284","author":"P Lakhani","year":"2017","journal-title":"Radiology"},{"issue":"2","key":"pone.0261307.ref008","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal Loss for Dense Object Detection","volume":"42","author":"TY Lin","year":"2020","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"pone.0261307.ref009","first-page":"5156416","article-title":"Improving Accuracy of Lung Nodule Classification Using Deep Learning with Focal Loss","volume":"2019","author":"GS Tran","year":"2019","journal-title":"J Healthc Eng"},{"issue":"3","key":"pone.0261307.ref010","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1093\/jamia\/ocaa280","article-title":"FLANNEL (Focal Loss bAsed Neural Network EnsembLe) for COVID-19 detection","volume":"28","author":"Z Qiao","year":"2021","journal-title":"Journal of the American Medical Informatics Association"},{"key":"pone.0261307.ref011","first-page":"1","article-title":"Ensemble Methods in Machine Learning","author":"TG Dietterich","year":"2000","journal-title":"Mult. Classif. Syst. 1857"},{"key":"pone.0261307.ref012","first-page":"1106","article-title":"ImageNet Classification with Deep Convolutional Neural Networks","author":"A Krizhevsky","year":"2012","journal-title":"NIPS"},{"key":"pone.0261307.ref013","doi-asserted-by":"crossref","unstructured":"Yadav O, Passi K, Jain CK. Using Deep Learning to Classify X-ray Images of Potential Tuberculosis Patients. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2018, pp. 2368\u20132375, doi: 10.1109\/BIBM.2018.8621525","DOI":"10.1109\/BIBM.2018.8621525"},{"key":"pone.0261307.ref014","doi-asserted-by":"crossref","first-page":"115041","DOI":"10.1109\/ACCESS.2020.3003810","article-title":"Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays","volume":"8","author":"S Rajaraman","year":"2020","journal-title":"IEEE Access"},{"key":"pone.0261307.ref015","doi-asserted-by":"crossref","first-page":"e8693","DOI":"10.7717\/peerj.8693","article-title":"Detection and visualization of abnormality in chest radiographs using modality-specific convolutional neural network ensembles","volume":"8","author":"S Rajaraman","year":"2020","journal-title":"PeerJ"},{"key":"pone.0261307.ref016","doi-asserted-by":"crossref","first-page":"27318","DOI":"10.1109\/ACCESS.2020.2971257","article-title":"Modality-specific deep learning model ensembles toward improving TB detection in chest radiographs","volume":"8","author":"S Rajaraman","year":"2020","journal-title":"IEEE Access"},{"issue":"2","key":"pone.0261307.ref017","doi-asserted-by":"crossref","first-page":"577","DOI":"10.1109\/TMI.2013.2290491","article-title":"Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration","volume":"33","author":"S Candemir","year":"2014","journal-title":"IEEE Trans Med Imaging"},{"key":"pone.0261307.ref018","first-page":"6105","article-title":"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.","author":"M Tan","year":"2019","journal-title":"ICML"},{"issue":"6","key":"pone.0261307.ref019","first-page":"475","article-title":"Two public chest X-ray datasets for computer-aided screening of pulmonary diseases","volume":"4","author":"S Jaeger","year":"2014","journal-title":"Quant Imaging Med Surg"},{"issue":"3","key":"pone.0261307.ref020","first-page":"234","article-title":"U-Net: Convolutional Networks for Biomedical Image Segmentation.","author":"O Ronneberger","year":"2015","journal-title":"MICCAI"},{"key":"pone.0261307.ref021","unstructured":"Yakubovskiy P. Segmentation Models. 2019, GitHub repository. Available at: https:\/\/github.com\/qubvel\/segmentation_models. Retrieved March 03, 2021"},{"key":"pone.0261307.ref022","first-page":"379","article-title":"Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks","author":"SSM Salehi","year":"2017","journal-title":"MLMI@MICCAI"},{"key":"pone.0261307.ref023","first-page":"683","article-title":"A Novel Focal Tversky Loss Function With Improved Attention U-Net for Lesion Segmentation","author":"N Abraham","year":"2019","journal-title":"ISBI"},{"issue":"10","key":"pone.0261307.ref024","doi-asserted-by":"crossref","first-page":"1715","DOI":"10.3390\/app8101715","article-title":"Visualization and Interpretation of Convolutional Neural Network Predictions in Detecting Pneumonia in Pediatric Chest Radiographs","volume":"8","author":"S Rajaraman","year":"2018","journal-title":"Applied Sciences"},{"issue":"4","key":"pone.0261307.ref025","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1017\/thg.2017.28","article-title":"CSOLNP: Numerical Optimization Engine for Solving Non-linearly Constrained Problems","volume":"20","author":"M Zahery","year":"2017","journal-title":"Twin Res Hum Genet"},{"key":"pone.0261307.ref026","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1016\/j.neunet.2020.09.017","article-title":"Deconvolutional neural network for image super-resolution","volume":"132","author":"F Cao","year":"2020","journal-title":"Neural Netw."},{"issue":"6","key":"pone.0261307.ref027","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1109\/TNSRE.2019.2913142","article-title":"Multiclass Support Matrix Machines by Maximizing the Inter-Class Margin for Single Trial EEG Classification","volume":"27","author":"I Razzak","year":"2019","journal-title":"IEEE Trans Neural Syst Rehabil Eng"},{"key":"pone.0261307.ref028","doi-asserted-by":"crossref","first-page":"5984","DOI":"10.1109\/TIP.2021.3089942","article-title":"Delving Deep Into Label Smoothing","volume":"30","author":"CB Zhang","year":"2021","journal-title":"IEEE Trans Image Process"},{"key":"pone.0261307.ref029","article-title":"Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification","author":"G Liang","year":"2020","journal-title":"CoRR abs\/2009.04057"},{"key":"pone.0261307.ref030","doi-asserted-by":"crossref","first-page":"100571","DOI":"10.1016\/j.imu.2021.100571","article-title":"UMS-Rep: Unified modality-specific representation for efficient medical image analysis","volume":"24","author":"G Zamzmi","year":"2021","journal-title":"Informatics in Medicine Unlocked."},{"key":"pone.0261307.ref031","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1186\/s12864-019-6413-7","article-title":"The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation","volume":"21","author":"D Chicco","year":"2020","journal-title":"BMC Genomics"},{"key":"pone.0261307.ref032","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618\u2013626, doi: 10.1109\/ICCV.2017.74","DOI":"10.1109\/ICCV.2017.74"},{"key":"pone.0261307.ref033","doi-asserted-by":"crossref","unstructured":"Ferreira JR, Cardenas DAC, Moreno RA, de F\u00e1tima de S\u00e1 Rebelo M, Krieger JE, Gutierrez MA. Multi-View Ensemble Convolutional Neural Network to Improve Classification of Pneumonia in Low Contrast Chest X-Ray Images. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2020, pp. 1238\u20131241, doi: 10.1109\/EMBC44109.2020.9176517","DOI":"10.1109\/EMBC44109.2020.9176517"},{"key":"pone.0261307.ref034","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1038\/s41746-020-0273-z","article-title":"Automated abnormality classification of chest radiographs using deep convolutional neural networks","volume":"3","author":"YX Tang","year":"2020","journal-title":"npj Digit. Med."},{"issue":"12","key":"pone.0261307.ref035","doi-asserted-by":"crossref","first-page":"102798","DOI":"10.1016\/j.scs.2021.102798","article-title":"Recognizing suspect and predicting the spread of contagion based on mobile phone location data (counteract): a system of identifying covid-19 infectious and hazardous sites, detecting disease outbreaks based on the internet of things, edge computing, and artificial intelligence","volume":"69","author":"H Ghayvat","year":"2021","journal-title":"Sustainable Cities and Society"},{"key":"pone.0261307.ref036","doi-asserted-by":"crossref","DOI":"10.1016\/j.scs.2020.102589","article-title":"Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey,","volume":"65","author":"S Bhattacharya","year":"2021","journal-title":"Sustainable Cities and Society"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0261307","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,30]],"date-time":"2021-12-30T18:29:24Z","timestamp":1640888964000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0261307"}},"subtitle":[],"editor":[{"given":"Thippa Reddy","family":"Gadekallu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,12,30]]},"references-count":36,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12,30]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0261307","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,30]]}}}