{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T21:17:44Z","timestamp":1770326264316,"version":"3.49.0"},"reference-count":52,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,9,26]],"date-time":"2021-09-26T00:00:00Z","timestamp":1632614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>The extended utilization of digitized Whole Slide Images is transforming the workflow of traditional clinical histopathology to the digital era. The ongoing transformation has demonstrated major potentials towards the exploitation of Machine Learning and Deep Learning techniques as assistive tools for specialized medical personnel. While the performance of the implemented algorithms is continually boosted by the mass production of generated Whole Slide Images and the development of state-of the-art deep convolutional architectures, ensemble models provide an additional methodology towards the improvement of the prediction accuracy. Despite the earlier belief related to deep convolutional networks being treated as black boxes, important steps for the interpretation of such predictive models have also been proposed recently. However, this trend is not fully unveiled for the ensemble models. The paper investigates the application of an explanation scheme for ensemble classifiers, while providing satisfactory classification results of histopathology breast and colon cancer images in terms of accuracy. The results can be interpreted by the hidden layers\u2019 activation of the included subnetworks and provide more accurate results than single network implementations.<\/jats:p>","DOI":"10.3390\/a14100278","type":"journal-article","created":{"date-parts":[[2021,9,27]],"date-time":"2021-09-27T01:59:47Z","timestamp":1632707987000},"page":"278","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["Ensembling EfficientNets for the Classification and Interpretation of Histopathology Images"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9234-0069","authenticated-orcid":false,"given":"Athanasios","family":"Kallipolitis","sequence":"first","affiliation":[{"name":"Department of Digital Systems, University of Piraeus, 18534 Piraeus, Greece"}]},{"given":"Kyriakos","family":"Revelos","sequence":"additional","affiliation":[{"name":"251 Hellenic Air Force and Veterans General Hospital, 11525 Athens, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2860-399X","authenticated-orcid":false,"given":"Ilias","family":"Maglogiannis","sequence":"additional","affiliation":[{"name":"Department of Digital Systems, University of Piraeus, 18534 Piraeus, Greece"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1590\/0100-3984.2019.0049","article-title":"Artificial intelligence, machine learning, computer-aided diagnosis, and radiomics: Advances in imaging towards to precision medicine","volume":"52","author":"Santos","year":"2019","journal-title":"Radiol. Bras."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"24454","DOI":"10.1038\/srep24454","article-title":"Computer-Aided Diagnosis with Deep Learning Architecture: Applications to Breast Lesions in US Images and Pulmonary Nodules in CT Scans","volume":"6","author":"Cheng","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e13930","DOI":"10.2196\/13930","article-title":"Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review","volume":"5","author":"Chan","year":"2019","journal-title":"JMIR Med. Educ."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.ejca.2019.06.012","article-title":"Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images","volume":"118","author":"Hekler","year":"2019","journal-title":"Eur. J. Cancer"},{"key":"ref_5","first-page":"421","article-title":"Machine learning outperforms human experts in MRI pattern analysis of muscular dystrophies","volume":"94","author":"Morrow","year":"2020","journal-title":"Neurol. Mar."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"954","DOI":"10.1038\/s41591-019-0447-x","article-title":"End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography","volume":"25","author":"Ardila","year":"2019","journal-title":"Nat. Med."},{"key":"ref_7","first-page":"633","article-title":"Initiative for the Alzheimers Disease Neuroimaging. Manifold learning of brain MRIs by deep learning","volume":"16","author":"Brosch","year":"2013","journal-title":"Med. Image Comput. Comput. Assist. Interv."},{"key":"ref_8","unstructured":"Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., and Shpanskaya, K. (2017). CheXNet: Radiologist-Level Pneumonia Detection on Chest X-rays with Deep Learning. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"100128","DOI":"10.1016\/j.patter.2020.100128","article-title":"Virtual Monoenergetic CT Imaging via Deep Learning","volume":"1","author":"Cong","year":"2020","journal-title":"Patterns"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"831","DOI":"10.1016\/j.gie.2020.04.039","article-title":"Deep learning for wireless capsule endoscopy: A systematic review and meta-analysis","volume":"92","author":"Soffer","year":"2020","journal-title":"Gastrointest. Endosc."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"182060","DOI":"10.1109\/ACCESS.2019.2958264","article-title":"Infrared Thermal Imaging-Based Crack Detection Using Deep Learning","volume":"7","author":"Yang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1016\/j.cmpb.2014.12.001","article-title":"Enhancing classification accuracy utilizing globules and dots features in digital dermoscopy","volume":"118","author":"Maglogiannis","year":"2015","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/TITB.2006.888702","article-title":"Radial basis function neural networks classification for the recognition of idiopathic pulmonary fibrosis in microscopic images","volume":"12","author":"Maglogiannis","year":"2008","journal-title":"IEEE Trans. Inf. Technol. Biomed. A Publ. IEEE Eng. Med. Biol. Soc."},{"key":"ref_14","first-page":"246","article-title":"Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network","volume":"16","author":"Prasoon","year":"2013","journal-title":"Med. Image Comput. Comput. Assist. Interv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"713","DOI":"10.21037\/atm.2020.02.44","article-title":"A review of the application of deep learning in medical image classification and segmentation","volume":"8","author":"Cai","year":"2020","journal-title":"Ann. Transl. Med."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Han, C., Hayashi, H., Rundo, L., Araki, R., Shimoda, W., Muramatsu, S., Furukawa, Y., Mauri, G., and Nakayama, H. (2018, January 4\u20137). GAN-based synthetic brain MR image generation. Proceedings of the IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.","DOI":"10.1109\/ISBI.2018.8363678"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1007\/s00138-020-01060-x","article-title":"Deep learning in medical image registration: A survey","volume":"31","author":"Haskins","year":"2020","journal-title":"Mach. Vis. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1134\/S1054661811020696","article-title":"Medical image registration based on SURF detector","volume":"21","author":"Lukashevich","year":"2011","journal-title":"Pattern Recognit. Image Anal."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.compmedimag.2018.08.010","article-title":"Deep learning nuclei detection: A simple approach can deliver state-of-the-art results","volume":"70","author":"Homeyer","year":"2018","journal-title":"Comput. Med. Imaging Graph."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kucharski, D., Kleczek, P., Jaworek-Korjakowska, J., Dyduch, G., and Gorgon, M. (2020). Semi-Supervised Nests of Melanocytes Segmentation Method Using Convolutional Autoencoders. Sensors, 20.","DOI":"10.3390\/s20061546"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"100089","DOI":"10.1016\/j.patter.2020.100089","article-title":"Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential","volume":"1","author":"Tschuchnig","year":"2020","journal-title":"Patterns"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kallipolitis, A., Stratigos, A., Zarras, A., and Maglogiannis, I. (2020, January 27\u201329). Fully Connected Visual Words for the Classification of Skin Cancer Confocal Images. Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Valletta, Malta.","DOI":"10.5220\/0009328808530858"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kallipolitis, A., and Maglogiannis, I. (2019, January 23\u201327). Creating Visual Vocabularies for the Retrieval and Classification of Histopathology Images. Proceedings of the 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany.","DOI":"10.1109\/EMBC.2019.8857126"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"L\u00f3pez-Abente, G., Mispireta, S., and Poll\u00e1n, M. (2014). Breast and prostate cancer: An analysis of common epidemiological features in mortality trends in Spain. BMC Cancer, 14.","DOI":"10.1186\/1471-2407-14-874"},{"key":"ref_25","unstructured":"Stewart, B.W., and Wild, C.P. (2017). Chapter 1.1: The Global and Regional Burden of Cancer, World Health Organization. The International Agency for Research on Cancer."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1007\/s11517-006-0079-4","article-title":"Neural network-based diagnostic and prognostic estimations in breast cancer microscopic instances","volume":"44","author":"Anagnostopoulos","year":"2006","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10916-015-0225-3","article-title":"An Advanced Image Analysis Tool for the Quantification and Characterization of Breast Cancer in Microscopy Images","volume":"39","author":"Goudas","year":"2015","journal-title":"J. Med. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Alinsaif, S., and Lang, J. (2020, January 13\u201315). Histological Image Classification using Deep Features and Transfer Learning. Proceedings of the 17th Conference on Computer and Robot Vision (CRV), Ottawa, ON, Canada.","DOI":"10.1109\/CRV50864.2020.00022"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_30","unstructured":"Kassani, S.H., Kassani, P.H., Wesolowski, M.J., Schneider, K.A., and Deters, R. (2019, January 4\u20136). Classification of histopathological biopsy images using ensemble of deep learning networks. Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering (CASCON 19), Markham, ON, Canada."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Livieris, I.E., Kanavos, A., Tampakas, V., and Pintelas, P. (2019). A Weighted Voting Ensemble Self-Labeled Algorithm for the Detection of Lung Abnormalities from X-Rays. Algorithms, 12.","DOI":"10.3390\/a12030064"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1662","DOI":"10.1109\/TIP.2020.3046875","article-title":"Loss-Based Attention for Interpreting Image-Level Prediction of Convolutional Neural Networks","volume":"30","author":"Shi","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_33","unstructured":"Tan, M., and Le, Q.V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chollet, F. (2017, January 21\u201326). Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.195"},{"key":"ref_35","unstructured":"Hu, J., Shen, L., Sun, G., and Albanie, S. (2018, January 18\u201323). Squeeze-and-Excitation Networks. Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, Salt Lake, UT, USA."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2016, January 12\u201317). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_37","unstructured":"Kaiming, H., Xiangyu, Z., Shaoqing, R., and Jian, S. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.bspc.2018.08.007","article-title":"Heartbeat classification fusing temporal and morphological information of ECGs via ensemble of classifiers","volume":"47","author":"Novo","year":"2019","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Hong, S., Wu, M., Zhou, Y., Wang, Q., Shang, J., Li, H., and Xie, J. (2017, January 24\u201327). ENCASE: An ENsemble ClASsifiEr for ECG classification using expert features and deep neural networks. Proceedings of the Computing in Cardiology (CinC), Rennes, France.","DOI":"10.22489\/CinC.2017.178-245"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"100864","DOI":"10.1016\/j.mex.2020.100864","article-title":"Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data","volume":"7","author":"Gessert","year":"2020","journal-title":"MethodsX"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s12530-019-09324-2","article-title":"On ensemble techniques of weight-constrained neural networks","volume":"12","author":"Livieris","year":"2021","journal-title":"Evol. Syst."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"3429","DOI":"10.1109\/TMI.2020.2995518","article-title":"Semi-Supervised Medical Image Classification with Relation-Driven Self-Ensembling Model","volume":"39","author":"Liu","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_43","first-page":"77","article-title":"Classifier combinations: Implementations and theoretical issues","volume":"Volume 1857","author":"Lam","year":"2000","journal-title":"Proceedings of the First International Workshop on Multiple Classifier Systems of Lecture Notes in Computer Science, MCS 2000"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wu, Y., Liu, L., Xie, Z., Bae, J., Chow, K., and Wei, W. (2020). Promoting High Diversity Ensemble Learning with EnsembleBench. arXiv.","DOI":"10.1109\/CogMI50398.2020.00034"},{"key":"ref_45","unstructured":"Tran, L., Veeling, B.S., Roth, K., Swiatkowski, J., Dillon, J.V., Snoek, J., Mandt, S., Salimans, T., Nowozin, S., and Jenatton, R. (2020). Hydra: Preserving Ensemble Diversity for Model Distillation. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Pintelas, E., Livieris, I.E., and Pintelas, P. (2020). A Grey-Box Ensemble Model Exploiting Black-Box Accuracy and White-Box Intrinsic Interpretability. Algorithms, 13.","DOI":"10.3390\/a13010017"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref_48","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (July, January 26). Learning Deep Features for Discriminative Localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L., Li, \u039a., and Li, F.-F. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1455","DOI":"10.1109\/TBME.2015.2496264","article-title":"A Dataset for Breast Cancer Histopathological Image Classification","volume":"63","author":"Spanhol","year":"2016","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_51","unstructured":"Kather, J.N., Halama, N., and Marx, A. (2018). 100,000 histological images of human colorectal cancer and healthy tissue (Version v0.1). Zenodo, 5281."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.media.2019.05.010","article-title":"BACH: Grand challenge on breast cancer histology images","volume":"56","author":"Aresta","year":"2019","journal-title":"Med. Image Anal."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/10\/278\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:05:20Z","timestamp":1760166320000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/10\/278"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,26]]},"references-count":52,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["a14100278"],"URL":"https:\/\/doi.org\/10.3390\/a14100278","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,26]]}}}