{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T12:29:35Z","timestamp":1792067375285,"version":"4.4.0"},"reference-count":41,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,3,3]],"date-time":"2021-03-03T00:00:00Z","timestamp":1614729600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010790","name":"Erasmus+","doi-asserted-by":"publisher","award":["Key Action 1 \u2013 Student Mobility for Traineeship, 2019\/2020"],"award-info":[{"award-number":["Key Action 1 \u2013 Student Mobility for Traineeship, 2019\/2020"]}],"id":[{"id":"10.13039\/501100010790","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The recognition of COVID-19 infection from X-ray images is an emerging field in the learning and computer vision community. Despite the great efforts that have been made in this field since the appearance of COVID-19 (2019), the field still suffers from two drawbacks. First, the number of available X-ray scans labeled as COVID-19-infected is relatively small. Second, all the works that have been carried out in the field are separate; there are no unified data, classes, and evaluation protocols. In this work, based on public and newly collected data, we propose two X-ray COVID-19 databases, which are three-class COVID-19 and five-class COVID-19 datasets. For both databases, we evaluate different deep learning architectures. Moreover, we propose an Ensemble-CNNs approach which outperforms the deep learning architectures and shows promising results in both databases. In other words, our proposed Ensemble-CNNs achieved a high performance in the recognition of COVID-19 infection, resulting in accuracies of 100% and 98.1% in the three-class and five-class scenarios, respectively. In addition, our approach achieved promising results in the overall recognition accuracy of 75.23% and 81.0% for the three-class and five-class scenarios, respectively. We make our databases of COVID-19 X-ray scans publicly available to encourage other researchers to use it as a benchmark for their studies and comparisons.<\/jats:p>","DOI":"10.3390\/s21051742","type":"journal-article","created":{"date-parts":[[2021,3,3]],"date-time":"2021-03-03T20:33:57Z","timestamp":1614803637000},"page":"1742","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":54,"title":["COVID-19 Recognition Using Ensemble-CNNs in Two New Chest X-ray Databases"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0965-4919","authenticated-orcid":false,"given":"Edoardo","family":"Vantaggiato","sequence":"first","affiliation":[{"name":"Department of Innovation Engineering, University of Salento, 73100 Lecce, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9473-044X","authenticated-orcid":false,"given":"Emanuela","family":"Paladini","sequence":"additional","affiliation":[{"name":"Department of Innovation Engineering, University of Salento, 73100 Lecce, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5077-4862","authenticated-orcid":false,"given":"Fares","family":"Bougourzi","sequence":"additional","affiliation":[{"name":"IEMN UMR CNRS 8520, Universit\u00e9 Polytechnique Hauts de France, UPHF, 59300 Famars, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1073-2390","authenticated-orcid":false,"given":"Cosimo","family":"Distante","sequence":"additional","affiliation":[{"name":"Department of Innovation Engineering, University of Salento, 73100 Lecce, Italy"},{"name":"Institute of Applied Sciences and Intelligent Systems, National Research Council of Italy, 73100 Lecce, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9092-735X","authenticated-orcid":false,"given":"Abdenour","family":"Hadid","sequence":"additional","affiliation":[{"name":"IEMN UMR CNRS 8520, Universit\u00e9 Polytechnique Hauts de France, UPHF, 59300 Famars, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8750-1905","authenticated-orcid":false,"given":"Abdelmalik","family":"Taleb-Ahmed","sequence":"additional","affiliation":[{"name":"IEMN UMR CNRS 8520, Universit\u00e9 Polytechnique Hauts de France, UPHF, 59300 Famars, France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1213","DOI":"10.1007\/s10489-020-01888-w","article-title":"COVIDetectioNet: COVID-19 diagnosis system based on X-ray images using features selected from pre-learned deep features ensemble","volume":"51","author":"Turkoglu","year":"2020","journal-title":"Appl. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"262","DOI":"10.7326\/M20-1495","article-title":"Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction\u2014Based SARS-CoV-2 Tests by Time Since Exposure","volume":"173","author":"Kucirka","year":"2020","journal-title":"Ann. Intern. Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"E113","DOI":"10.1148\/radiol.2020200527","article-title":"Essentials for Radiologists on COVID-19: An Update\u2014Radiology Scientific Expert Panel","volume":"296","author":"Kanne","year":"2020","journal-title":"Radiology"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"427","DOI":"10.3389\/fmed.2020.00427","article-title":"Deep learning-based decision-tree classifier for COVID-19 diagnosis from chest X-ray imaging","volume":"7","author":"Yoo","year":"2020","journal-title":"Front. Med."},{"key":"ref_5","first-page":"1","article-title":"Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest X-ray images","volume":"10","author":"Wang","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rahman, T., Chowdhury, M.E.H., Khandakar, A., Islam, K.R., Islam, K.F., Mahbub, Z.B., Kadir, M.A., and Kashem, S. (2020). Transfer Learning with Deep Convolutional Neural Network (CNN) for Pneumonia Detection Using Chest X-ray. Appl. Sci., 10.","DOI":"10.3390\/app10093233"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.patrec.2021.01.010","article-title":"Deep learning for real-time semantic segmentation: Application in ultrasound imaging","volume":"144","author":"Ouahabi","year":"2021","journal-title":"Pattern Recognit. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"8889023","DOI":"10.1155\/2020\/8889023","article-title":"Artificial intelligence-based classification of chest X-ray images into COVID-19 and other infectious diseases","volume":"2020","author":"Sharma","year":"2020","journal-title":"Int. J. Biomed. Imaging"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Karim, M., D\u00f6hmen, T., Rebholz-Schuhmann, D., Decker, S., Cochez, M., and Beyan, O. (2020). Deepcovidexplainer: Explainable covid-19 predictions based on chest X-ray images. arXiv.","DOI":"10.1109\/BIBM49941.2020.9313304"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Hammoudi, K., Benhabiles, H., Melkemi, M., Dornaika, F., Arganda-Carreras, I., Collard, D., and Scherpereel, A. (2020). Deep Learning on Chest X-ray Images to Detect and Evaluate Pneumonia Cases at the Era of COVID-19. arXiv.","DOI":"10.1007\/s10916-021-01745-4"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"113459","DOI":"10.1016\/j.eswa.2020.113459","article-title":"Fusion Transformed Deep and Shallow features (FTDS) for Image-Based Facial Expression Recognition","volume":"156","author":"Bougourzi","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1479","DOI":"10.1049\/iet-ipr.2018.6235","article-title":"Fusion of transformed shallow features for facial expression recognition","volume":"13","author":"Bougourzi","year":"2019","journal-title":"IET Image Process."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1007\/s13246-020-00865-4","article-title":"Covid-19: Automatic detection from X-ray images utilizing transfer learning with convolutional neural networks","volume":"43","author":"Apostolopoulos","year":"2020","journal-title":"Phys. Eng. Sci. Med."},{"key":"ref_14","unstructured":"Hemdan, E.E.D., Shouman, M.A., and Karar, M.E. (2020). Covidx-net: A framework of deep learning classifiers to diagnose covid-19 in X-ray images. arXiv."},{"key":"ref_15","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA. Issue 1.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201323). Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_22","unstructured":"Mangal, A., Kalia, S., Rajgopal, H., Rangarajan, K., Namboodiri, V., Banerjee, S., and Arora, C. (2020). CovidAID: COVID-19 Detection Using Chest X-ray. arXiv."},{"key":"ref_23","unstructured":"Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., and Shpanskaya, K. (2017). Chexnet: Radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2017, January 21\u201326). Chestx-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.369"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"Imagenet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kira, K., and Rendell, L.A. (1992). A practical approach to feature selection. Machine Learning Proceedings 1992, Elsevier.","DOI":"10.1016\/B978-1-55860-247-2.50037-1"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"15364","DOI":"10.1038\/s41598-020-71294-2","article-title":"COVID-19 image classification using deep features and fractional-order marine predators algorithm","volume":"10","author":"Sahlol","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cohen, J.P., Morrison, P., Dao, L., Roth, K., Duong, T.Q., and Ghassemi, M. (2020). Covid-19 image data collection: Prospective predictions are the future. arXiv.","DOI":"10.59275\/j.melba.2020-48g7"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1016\/j.cell.2018.02.010","article-title":"Identifying medical diagnoses and treatable diseases by image-based deep learning","volume":"172","author":"Kermany","year":"2018","journal-title":"Cell"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"e180041","DOI":"10.1148\/ryai.2019180041","article-title":"Augmenting the National Institutes of Health chest radiograph dataset with expert annotations of possible pneumonia","volume":"1","author":"Shih","year":"2019","journal-title":"Radiol. Artif. Intell."},{"key":"ref_32","unstructured":"Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., and Shpanskaya, K. (February, January 27). Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_33","first-page":"475","article-title":"Two public chest X-ray datasets for computer-aided screening of pulmonary diseases","volume":"4","author":"Jaeger","year":"2014","journal-title":"Quant. Imaging Med. Surg."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Xie, S., Girshick, R., Doll\u00e1r, P., Tu, Z., and He, K. (2017, January 21\u201326). Aggregated residual transformations for deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.634"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Goyal, P., Girshick, R., He, K., and Doll\u00e1r, P. (2017, January 21\u201326). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Honolulu, HI, USA.","DOI":"10.1109\/ICCV.2017.324"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"012043","DOI":"10.1088\/1757-899X\/428\/1\/012043","article-title":"Age Classification Using Convolutional Neural Networks with the Multi-class Focal Loss","volume":"428","author":"Liu","year":"2018","journal-title":"IOP Conf. Ser. Mater. Sci. Eng."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bendjoudi, I., Vanderhaegen, F., Hamad, D., and Dornaika, F. (2020). Multi-label, multi-task CNN approach for context-based emotion recognition. Inf. Fusion.","DOI":"10.1016\/j.inffus.2020.11.007"},{"key":"ref_39","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019, January 8\u201314). Pytorch: An imperative style, high-performance deep learning library. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_40","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_41","unstructured":"Petsiuk, V., Das, A., and Saenko, K. (2018). RISE: Randomized Input Sampling for Explanation of Black-box Models. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/5\/1742\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:32:01Z","timestamp":1760160721000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/5\/1742"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,3]]},"references-count":41,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21051742"],"URL":"https:\/\/doi.org\/10.3390\/s21051742","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,3]]}}}