{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T19:06:26Z","timestamp":1772910386993,"version":"3.50.1"},"reference-count":57,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,6,7]],"date-time":"2024-06-07T00:00:00Z","timestamp":1717718400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research, Development, and Innovation Fund of Hungary","award":["TKP2021-NKTA-34"],"award-info":[{"award-number":["TKP2021-NKTA-34"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Chest X-rays are vital in healthcare for diagnosing various conditions due to their low Radiation exposure, widespread availability, and rapid interpretation. However, their interpretation requires specialized expertise, which can limit scalability and delay diagnoses. This study addresses the multi-label classification challenge of chest X-ray images using the Chest X-ray14 dataset. We propose a novel online ensemble technique that differs from previous penalty-based methods by focusing on combining individual model losses with the overall ensemble loss. This approach enhances interaction and feedback among models during training. Our method integrates multiple pre-trained CNNs using strategies like combining CNNs through an additional fully connected layer and employing a label-weighted average for outputs. This multi-layered approach leverages the strengths of each model component, improving classification accuracy and generalization. By focusing solely on image data, our ensemble model addresses the challenges posed by null vectors and diverse pathologies, advancing computer-aided radiology.<\/jats:p>","DOI":"10.3390\/make6020060","type":"journal-article","created":{"date-parts":[[2024,6,7]],"date-time":"2024-06-07T11:56:48Z","timestamp":1717761408000},"page":"1281-1297","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Advanced Multi-Label Image Classification Techniques Using Ensemble Methods"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-0625-3959","authenticated-orcid":false,"given":"Tam\u00e1s","family":"Katona","sequence":"first","affiliation":[{"name":"Doctoral School of Informatics, University of Debrecen, 4028 Debrecen, Hungary"},{"name":"Department of Data Science and Visualization, Faculty of Informatics, University of Debrecen, 4028 Debrecen, Hungary"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3059-0243","authenticated-orcid":false,"given":"G\u00e1bor","family":"T\u00f3th","sequence":"additional","affiliation":[{"name":"Department of Laboratory Medicine, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8676-8787","authenticated-orcid":false,"given":"M\u00e1ty\u00e1s","family":"Petr\u00f3","sequence":"additional","affiliation":[{"name":"Department of Radiology, Medical Imaging Institute, Faculty of Medicine, University of Debrecen, 4032 Debrecen, Hungary"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4405-2040","authenticated-orcid":false,"given":"Bal\u00e1zs","family":"Harangi","sequence":"additional","affiliation":[{"name":"Department of Data Science and Visualization, Faculty of Informatics, University of Debrecen, 4028 Debrecen, Hungary"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1148\/radiol.2015150921","article-title":"The U.S. Radiologist Workforce: An Analysis of Temporal and Geographic Variation by Using Large National Datasets","volume":"279","author":"Rosenkrantz","year":"2016","journal-title":"Radiology"},{"key":"ref_2","first-page":"6","article-title":"Diagnostic radiology in Liberia: A country report","volume":"1","author":"Ali","year":"2015","journal-title":"J. Glob. Radiol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2017, January 21\u201326). Chestx-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.369"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1109\/TBME.1972.324115","article-title":"Automated Radiographic Diagnosis via Feature Extraction and Classification of Cardiac Size and Shape Descriptors","volume":"BME-19","author":"Kruger","year":"1972","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1148\/81.2.185","article-title":"The Coding of Roentgen Images for Computer Analysis as Applied to Lung Cancer","volume":"81","author":"Lodwick","year":"1963","journal-title":"Radiology"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1148\/83.6.1029","article-title":"Automated Computer Analysis of Radiographic Images","volume":"83","author":"Meyers","year":"1964","journal-title":"Radiology"},{"key":"ref_7","first-page":"94","article-title":"Machine learning approaches in medical image analysis: From detection to diagnosis","volume":"476","year":"2016","journal-title":"Med. Image Anal."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., and Shpanskaya, K. (February, January 27). CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. Proceedings of the AAAI Conference on Artificial Intelligence 2019, Honolulu, HI, USA.","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., and Summers, R.M. (2018, January 18\u201322). TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00943"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/978-3-030-00919-9_29","article-title":"Attention-guided curriculum learning for weakly supervised classification and localization of thoracic diseases on chest radiographs","volume":"Volume 1","author":"Tang","year":"2018","journal-title":"Proceedings of the International Workshop on Machine Learning in Medical Imaging (MLMI 2018)"},{"key":"ref_11","unstructured":"Yao, L., Prosky, J., Poblenz, E., Covington, B., and Lyman, K. (2018). Weakly supervised medical diagnosis and localization from multiple resolutions. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1016\/j.patrec.2018.10.027","article-title":"Multi-label chest X-ray image classification via category-wise residual attention learning","volume":"130","author":"Guan","year":"2020","journal-title":"Pattern Recognit. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"101846","DOI":"10.1016\/j.media.2020.101846","article-title":"Triple attention learning for classification of 14 thoracic diseases using chest radiography","volume":"67","author":"Wang","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yan, C., Yao, J., Li, R., Xu, Z., and Huang, J. (September, January 29). Weakly Supervised Deep Learning for Thoracic Disease Classification and Localization on Chest X-rays. Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Washington DC, USA.","DOI":"10.1145\/3233547.3233573"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"6381","DOI":"10.1038\/s41598-019-42294-8","article-title":"Comparison of Deep Learning Approaches for Multi-Label Chest X-ray Classification","volume":"9","author":"Baltruschat","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Katona, T., and Antal, B. (2019, January 23\u201325). Automated analysis of radiology images using Convolutional Neural Networks. Proceedings of the 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA 2019), Dubrovnik, Croatia.","DOI":"10.1109\/ISPA.2019.8868764"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Katona, T., T\u00f3th, G., Petr\u00f3, M., and Harangi, B. (2024). Developing New Fully Connected Layers for Convolutional Neural Networks with Hyperparameter Optimization for Improved Multi-Label Image Classification. Mathematics, 12.","DOI":"10.3390\/math12060806"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1080\/095400996116785","article-title":"On combining artificial neural nets","volume":"8","author":"Sharkey","year":"1996","journal-title":"Connect. Sci."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sharkey, A.J.C., Sharkey, N.E., Gerecke, U., and Chandroth, G.O. (2000, January 21\u201323). The \u201cTest and Select\u201d Approach to Ensemble Combination. Proceedings of the Multiple Classifier Systems. Multiple Classifier Systems 2000, Calgiari, Italy. Lecture Notes in Computer Science.","DOI":"10.1007\/3-540-45014-9_3"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Dvornik, N., Mairal, J., and Schmid, C. (November, January 27). Diversity with Cooperation: Ensemble Methods for Few-Shot Classification. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00382"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1109\/TPAMI.2018.2848925","article-title":"Deep Metric Learning with BIER: Boosting Independent Embeddings Robustly","volume":"42","author":"Opitz","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"982","DOI":"10.1109\/TPAMI.2019.2943860","article-title":"Nonlinear Regression via Deep Negative Correlation Learning","volume":"43","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"110358","DOI":"10.1109\/ACCESS.2019.2933670","article-title":"Ensemble Learners of Multiple Deep CNNs for Pulmonary Nodules Classification Using CT Images","volume":"7","author":"Zhang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1419","DOI":"10.1109\/TIFS.2018.2878542","article-title":"Ensemble of Multi-View Learning Classifiers for Cross-Domain Iris Presentation Attack Detection","volume":"14","author":"Kuehlkamp","year":"2019","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Maarouf, A.A., and Hachouf, F. (2022, January 17\u201318). Transfer Learning-based Ensemble Deep Learning for Road Cracks Detection. Proceedings of the 2022 International Conference on Advanced Aspects of Software Engineering (ICAASE), Constantine, Algeria.","DOI":"10.1109\/ICAASE56196.2022.9931581"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Khan, I.A., Sajeeb, A., and Fattah, S.A. (2020, January 17\u201319). An Automatic Ocular Disease Detection Scheme from Enhanced Fundus Images Based on Ensembling Deep CNN Networks. Proceedings of the 11th International Conference on Electrical and Computer Engineering, Dhaka, Bangladesh.","DOI":"10.1109\/ICECE51571.2020.9393050"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"36274","DOI":"10.1109\/ACCESS.2019.2903127","article-title":"Deep Learning-Based Classification Methods for Remote Sensing Images in Urban Built-Up Areas","volume":"7","author":"Li","year":"2019","journal-title":"IEEE Access"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1882","DOI":"10.1109\/JSTARS.2019.2915259","article-title":"Deep Learning Ensemble for Hyperspectral Image Classification","volume":"12","author":"Chen","year":"2019","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"6530","DOI":"10.1109\/TGRS.2019.2906883","article-title":"Hydra: An Ensemble of Convolutional Neural Networks for Geospatial Land Classification","volume":"57","author":"Minetto","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_30","first-page":"1","article-title":"Deep Ensemble CNN Method Based on Sample Expansion for Hyperspectral Image Classification","volume":"60","author":"Dong","year":"2022","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Alosaimi, N., and Alhichri, H. (2020, January 19\u201321). Fusion of CNN ensemble for Remote Sensing Scene Classification. Proceedings of the 2020 3rd International Conference on Computer Applications Information Security (ICCAIS), Riyadh, Saudi Arabia.","DOI":"10.1109\/ICCAIS48893.2020.9096721"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Elmannai, H., Saleh, H., Algarni, A.D., Mashal, I., Kwak, K.S., El-Sappagh, S., and Mostafa, S. (2022). Diagnosis Myocardial Infarction Based on Stacking Ensemble of Convolutional Neural Network. Electronics, 11.","DOI":"10.3390\/electronics11233976"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Mansoor, R., Shah, M.A., Khattak, H.A., Mussadiq, S., Rauf, H.T., and Ameer, Z. (2022). Detection of Diseases in Pandemic: A Predictive Approach Using Stack Ensembling on Multi-Modal Imaging Data. Electronics, 11.","DOI":"10.3390\/electronics11233974"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Mahanty, C., Kumar, R., Asteris, P.G., and Gandomi, A.H. (2021). COVID-19 Patient Detection Based on Fusion of Transfer Learning and Fuzzy Ensemble Models Using CXR Images. Appl. Sci., 11.","DOI":"10.3390\/app112311423"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.ins.2022.12.022","article-title":"Dynamic ensemble learning for multi-label classification","volume":"623","author":"Zhu","year":"2023","journal-title":"Inf. Sci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1016\/j.ins.2020.06.017","article-title":"Multi-label classification with weighted classifier selection and stacked ensemble","volume":"557","author":"Xia","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2151006","DOI":"10.1142\/S021800142151006X","article-title":"MLCE: A Multi-Label Crotch Ensemble Method for Multi-Label Classification","volume":"35","author":"Yao","year":"2021","journal-title":"Int. J. Pattern Recognit. Artif. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"911","DOI":"10.3390\/signals3040054","article-title":"Ensemble of Networks for Multilabel Classification","volume":"3","author":"Nanni","year":"2022","journal-title":"Signals"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.jbi.2018.08.006","article-title":"Skin lesion classification with ensembles of deep convolutional neural networks","volume":"86","author":"Harangi","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1016\/S0893-6080(99)00073-8","article-title":"Ensemble learning via negative correlation","volume":"12","author":"Liu","year":"1999","journal-title":"Neural Netw."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Harangi, B., Baran, A., Beregi-Kovacs, M., and Hajdu, A. (2023). Composing Diverse Ensembles of Convolutional Neural Networks by Penalization. Mathematics, 11.","DOI":"10.3390\/math11234730"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kufel, J., Biel\u00f3wka, M., Rojek, M., Mitr\u0119ga, A., Lewandowski, P., Cebula, M., Krawczyk, D., Biel\u00f3wka, M., Kondo\u0142, D., and Bargie\u0142-\u0141\u0105czek, K. (2023). Multi-Label Classification of Chest X-ray Abnormalities Using Transfer Learning Techniques. J. Pers. Med., 13.","DOI":"10.3390\/jpm13101426"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"Johnson","year":"2019","journal-title":"Sci. Data"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"E215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.-y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., and Horng, S. (2019). MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv.","DOI":"10.1038\/s41597-019-0322-0"},{"key":"ref_46","first-page":"274","article-title":"Atelectasis: Mechanisms, diagnosis and management","volume":"1","author":"Peroni","year":"2000","journal-title":"Paediatr. Respir. Rev."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"D\u00e4hnert, W. (2011). Radiology Review Manual, Wolters Kluwer Health\/Lippincott Williams Wilkins.","DOI":"10.1097\/MNM.0b013e32834b6e3d"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1183\/09031936.01.00213501","article-title":"Imaging of pneumonia: Trends and algorithms","volume":"18","author":"Franquet","year":"2001","journal-title":"Eur. Respir. J."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1513\/AnnalsATS.201412-576AS","article-title":"Differentiating Pneumothorax from the Common Radiographic Skinfold Artifact","volume":"12","author":"Kattea","year":"2015","journal-title":"Ann. Am. Thorac. Soc."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking the Inception Architecture for Computer Vision. arXiv.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_52","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. arXiv.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., and Doll\u00e1r, P. (2020). Designing Network Design Spaces. arXiv.","DOI":"10.1109\/CVPR42600.2020.01044"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017). Xception: Deep Learning with Depthwise Separable Convolutions. arXiv.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_56","unstructured":"Diederik, P.K., and Jimmy, B. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_57","unstructured":"Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/2\/60\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:55:15Z","timestamp":1760108115000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/2\/60"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,7]]},"references-count":57,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["make6020060"],"URL":"https:\/\/doi.org\/10.3390\/make6020060","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,7]]}}}