{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T09:06:37Z","timestamp":1777021597470,"version":"3.51.4"},"reference-count":53,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,2,7]],"date-time":"2021-02-07T00:00:00Z","timestamp":1612656000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The Coronavirus disease 2019 (COVID-19) has become one of the threats to the world. Computed tomography (CT) is an informative tool for the diagnosis of COVID-19 patients. Many deep learning approaches on CT images have been proposed and brought promising performance. However, due to the high complexity and non-transparency of deep models, the explanation of the diagnosis process is challenging, making it hard to evaluate whether such approaches are reliable. In this paper, we propose a visual interpretation architecture for the explanation of the deep learning models and apply the architecture in COVID-19 diagnosis. Our architecture designs a comprehensive interpretation about the deep model from different perspectives, including the training trends, diagnostic performance, learned features, feature extractors, the hidden layers, the support regions for diagnostic decision, and etc. With the interpretation architecture, researchers can make a comparison and explanation about the classification performance, gain insight into what the deep model learned from images, and obtain the supports for diagnostic decisions. Our deep model achieves the diagnostic result of 94.75%, 93.22%, 96.69%, 97.27%, and 91.88% in the criteria of accuracy, sensitivity, specificity, positive predictive value, and negative predictive value, which are 8.30%, 4.32%, 13.33%, 10.25%, and 6.19% higher than that of the compared traditional methods. The visualized features in 2-D and 3-D spaces provide the reasons for the superiority of our deep model. Our interpretation architecture would allow researchers to understand more about how and why deep models work, and can be used as interpretation solutions for any deep learning models based on convolutional neural network. It can also help deep learning methods to take a step forward in the clinical COVID-19 diagnosis field.<\/jats:p>","DOI":"10.3390\/e23020204","type":"journal-article","created":{"date-parts":[[2021,2,8]],"date-time":"2021-02-08T20:51:51Z","timestamp":1612817511000},"page":"204","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["An Interpretation Architecture for Deep Learning Models with the Application of COVID-19 Diagnosis"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5183-9074","authenticated-orcid":false,"given":"Yuchai","family":"Wan","sequence":"first","affiliation":[{"name":"Beijing Key Laboratory of Big Data Technology for Food Safety, School of Computer Science, Beijing Technology and Business University, Beijing 100048, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hongen","family":"Zhou","sequence":"additional","affiliation":[{"name":"Beijing Key Laboratory of Big Data Technology for Food Safety, School of Computer Science, Beijing Technology and Business University, Beijing 100048, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5502-2974","authenticated-orcid":false,"given":"Xun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Beijing Key Laboratory of Big Data Technology for Food Safety, School of Computer Science, Beijing Technology and Business University, Beijing 100048, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,7]]},"reference":[{"key":"ref_1","unstructured":"World Health Organization (2021, January 11). Novel Coronavirus (COVID-19) Situation. Available online: https:\/\/who.sprinklr.com."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/S0140-6736(20)32625-8","article-title":"Calling for pan-European commitment for rapid and sustained reduction in SARS-CoV-2 infections","volume":"397","author":"Priesemann","year":"2021","journal-title":"Lancet"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3389\/fphy.2020.00127","article-title":"Forecasting COVID-19","volume":"8","author":"Perc","year":"2020","journal-title":"Front. Phys."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1148\/radiol.2020200343","article-title":"Chest CT for typical 2019-nCoV pneumonia: Relationship to negative RT-PCR testing","volume":"296","author":"Xie","year":"2020","journal-title":"Radiology"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"22402","DOI":"10.1038\/s41598-020-80061-2","article-title":"A meta-analysis of accuracy and sensitivity of chest CT and RT-PCR in COVID-19 diagnosis","volume":"10","author":"Khatami","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.idcr.2020.e00905","article-title":"More Reliability of Suspicious Symptoms plus Chest CT-Scan than RT_PCR Test for the Diagnosis of COVID-19 in an 18-days-old Neonate","volume":"21","author":"Mahdavi","year":"2020","journal-title":"IDCases"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1016\/S0140-6736(20)30183-5","article-title":"Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China","volume":"395","author":"Huang","year":"2020","journal-title":"Lancet"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1001\/jama.2020.1585","article-title":"Clinical characteristics of 138 hospitalized patients with 2019 novel Coronavirus Infected pneumonia in wuhan, China","volume":"323","author":"Wang","year":"2020","journal-title":"JAMA"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1148\/radiol.2020201343","article-title":"Diagnostic Performance of CT and Reverse Transcriptase-Polymerase Chain Reaction for Coro navirus Disease 2019: A Meta-Analysis","volume":"296","author":"Kim","year":"2020","journal-title":"Radiology"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"105980","DOI":"10.1016\/j.rmed.2020.105980","article-title":"Diagnostic performance between CT and initial real-time RT-PCR for clinically suspected 2019 coronavirus disease (COVID-19) patients outside Wuhan, China","volume":"168","author":"He","year":"2020","journal-title":"Respir. Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1007\/s13246-020-00865-4","article-title":"Covid-19: Automatic detection from X-ray images utilizing transfer learning with convolutional neural networks","volume":"43","author":"Apostolopoulos","year":"2020","journal-title":"Phys. Eng. Sci. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"109581","DOI":"10.1109\/ACCESS.2020.3001973","article-title":"Artificial Intelligence and COVID-19: Deep Learning Approaches for Diagnosis and Treatment","volume":"8","author":"Jamshidi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"118869","DOI":"10.1109\/ACCESS.2020.3005510","article-title":"Weakly Supervised Deep Learning for COVID-19 Infection Detection and Classification from CT Images","volume":"8","author":"Hu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2606","DOI":"10.1109\/TMI.2020.2992546","article-title":"Diagnosis of Coronavirus Disease 2019 (COVID-19) with Structured Latent Multi-View Representation Learning","volume":"39","author":"Kang","year":"2020","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"155987","DOI":"10.1109\/ACCESS.2020.3018498","article-title":"Semi-supervised Network for Detection of COVID-19 in Chest CT Scans","volume":"8","author":"Mohammed","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","first-page":"1010","article-title":"COVID-19 detection and disease progression visualization: Deep learning on chest X-rays for classification and coarse localization","volume":"50","author":"Zebin","year":"2020","journal-title":"Appl. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2352","DOI":"10.1162\/neco_a_00990","article-title":"Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review","volume":"29","author":"Rawat","year":"2017","journal-title":"Neural Comput."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, Y., Cao, Y., Zha, Z., Zhang, J., and Xiong, Z. (2020, January 13\u201319). Deep Degradation Prior for Low-Quality Image Classification. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01106"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4420","DOI":"10.1109\/TGRS.2018.2818945","article-title":"3-D Deep Learning Approach for Remote Sensing Image Classification","volume":"56","author":"Hamida","year":"2018","journal-title":"IEEE Trans. Getsci. Remote Sens."},{"key":"ref_20","unstructured":"Peterson, J.C., Battleday, R.M., Griffiths, T.L., and Russakovsky, O. (November, January 27). Human uncertainty makes classification more robust. Proceedings of the International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Agustsson, E., Uijlings, J.R., and Ferrari, V. (2019, January 16\u201320). Interactive Full Image Segmentation by Considering All Regions Jointly. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01189"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ahn, J., Cho, S., and Kwak, S. (2019, January 16\u201320). Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00231"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"46335","DOI":"10.1109\/ACCESS.2020.2974101","article-title":"Feature Extraction and Analysis of Natural Language Processing for Deep Learning English Language","volume":"8","author":"Wang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_24","unstructured":"Young, T., Hazarika, D., Poria, S., and Cambria, E. (2018). Recent Trends in Deep Learning Based Natural Language Processing. arXiv, Available online: https:\/\/arxiv.org\/abs\/1708.02709v5."},{"key":"ref_25","unstructured":"Farooq, M., and Hafeez, A. (2020). Covid-resnet: A Deep Learning Framework for Screening of Covid19 from Radiographs. arXiv, Available online: https:\/\/arxiv.org\/abs\/2003.14395."},{"key":"ref_26","unstructured":"Hemdan, E.E., Shouman, M.A., and Karar, M.E. (2020). Covidx-net: A Framework of Deep Learning Classifiers to Diagnose Covid-19 in X-ray Images. arXiv, Available online: https:\/\/arxiv.org\/abs\/2003.11055."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"He, X., Yang, X., Zhang, S., Zhao, J., and Xie, P. (2020). Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT Scans. MedRxiv.","DOI":"10.1101\/2020.04.13.20063941"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Abdani, S.R., Zulkifley, M.A., and Zulkifley, N.H. (2020, January 17\u201318). A Lightweight Deep Learning Model for COVID-19 Detection. Proceedings of the IEEE Symposium on Industrial Electronics & Applications, TBD, Malaysia.","DOI":"10.1109\/ISIEA49364.2020.9188133"},{"key":"ref_29","first-page":"1","article-title":"Deep Learning Approaches for COVID-19 Detection Based on Chest X-ray Images","volume":"164","author":"Masood","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/RBME.2020.2987975","article-title":"Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation and Diagnosis for COVID-19","volume":"14","author":"Shi","year":"2020","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Guiot, J., Vaidyanathan, A., Deprez, L., Zerka, F., Danthine, D., Frix, A.-N., Thys, M., Henket, M., Canivet, G., and Mathieu, S. (2021). Development and Validation of an Automated Radiomic CT Signature for Detecting COVID-19. Diagnostics, 11.","DOI":"10.1101\/2020.04.28.20082966"},{"key":"ref_32","unstructured":"Alom, M., Rahman, M., Nasrin, M., Taha, T., and Asari, V. (2020). COVID_MTNet: COVID-19 Detection with Multi-Task Deep Learning Approaches. arXiv, Available online: https:\/\/arxiv.org\/abs\/2004.03747."},{"key":"ref_33","unstructured":"Zhi, Q., Austin, B., Glass, L., Xiao, C., and Sun, J. (2020). FLANNEL: Focal Loss Based Neural Network Ensemble for COVID-19 Detection. J. Am. Med. Inform. Assoc., 1\u20139."},{"key":"ref_34","unstructured":"Shan, F., Gao, Y., Wang, J., Shi, W., Shi, N., Han, M., Xue, Z., and Shi, Y. (2020). Lung Infection Quantification of COVID-19 in CT Images with Deep Learning. arXiv, Available online: https:\/\/arxiv.org\/abs\/2003.04655."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhang, P., Zhong, Y., Deng, Y., Tang, X., and Li, X. (2020). CoSinGAN: Learning COVID-19 Infection Segmentation from a Single Radiologi-cal Image. Diagnostics, 10.","DOI":"10.3390\/diagnostics10110901"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1148\/radiol.2020200905","article-title":"Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT","volume":"296","author":"Li","year":"2020","journal-title":"Radiology"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Jin, S., Wang, B., Xu, H., Luo, C., Wei, L., Zhao, W., Hou, X., Ma, W., Xu, Z., and Zheng, Z. (2020). AI-Assisted CT Imaging Analysis for COVID-19 Screening: Building and Deploying a Medical AI System in Four Weeks. MedRxiv.","DOI":"10.1101\/2020.03.19.20039354"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"e200075","DOI":"10.1148\/ryct.2020200075","article-title":"Serial quantitative chest CT assessment of COVID-19: Deep-Learning Approach","volume":"2","author":"Huang","year":"2020","journal-title":"Radiol. Cardiothorac. Imaging"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"e200082","DOI":"10.1148\/ryct.2020200082","article-title":"Longitudinal assessment of COVID-19 using a deep learning\u2013based quantitative CT pipeline: Illustration of two cases","volume":"2","author":"Gao","year":"2020","journal-title":"Radiol. Cardiothorac. Imaging"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Lipton, Z. (2016). The Mythos of Model Interpretability. Commun. ACM, 61.","DOI":"10.1145\/3233231"},{"key":"ref_41","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (July, January 26). Learning Deep Features for Discriminative Localization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Selvaraju, R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Proceedings of the International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1727","DOI":"10.1038\/s41598-018-20132-7","article-title":"Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs: A Deep Learning-Based Approach","volume":"8","author":"Tiulpin","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"110190","DOI":"10.1016\/j.chaos.2020.110190","article-title":"A Deep Learning and Grad-CAM based Color Visualization Approach for Fast Detection of COVID-19 Cases using Chest X-ray and CT-Scan Images","volume":"140","author":"Panwar","year":"2020","journal-title":"Chaos Solitons Fractals"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Ribeiro, M., Singh, S., and Guestrin, C. (2016, January 13\u201317). Why Should I Trust You? Explaining the Predictions of Any Classifier. Proceedings of the ACM SIGKDD, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Xu, Y., Jia, Z., Wang, L., Ai, Y., Zhang, F., Lai, M., and Chang, E. (2017). Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features. BMC Bioinform., 18.","DOI":"10.1186\/s12859-017-1685-x"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Rieke, J., Eitel, F., Weygandt, M., Haynes, J., and Ritter, K. (2018). Visualizing Convolutional Networks for MRI-based Diagnosis of Alzheimer\u2019s Disease. arXiv, Available online: https:\/\/arxiv.org\/abs\/1808.02874.","DOI":"10.1007\/978-3-030-02628-8_3"},{"key":"ref_48","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G. (2012, January 3\u20136). ImageNet classification with deep convolutional neural networks. Proceedings of the Conference and Workshop on Neural Information Processing Systems, Harrahs and Harveys, Lake Tahoe, NV, USA."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.jocs.2018.11.008","article-title":"Pathological brain detection based on AlexNet and transfer learning","volume":"30","author":"Lu","year":"2019","journal-title":"J. Comput. Sci."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"149","DOI":"10.3934\/mfc.2018008","article-title":"How convolutional neural networks see the world\u2014A survey of convolutional neural network visualization methods","volume":"1","author":"Qin","year":"2018","journal-title":"Math. Found. Comput."},{"key":"ref_52","unstructured":"Springenberg, J., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015). Striving for Simplicity: The All Convolutional Net. arXiv, Available online: https:\/\/arxiv.org\/abs\/1412.6806."},{"key":"ref_53","unstructured":"Ozbulak, U. (2021, February 03). PyTorch CNN Visualizations. Github, 2019. Available online: https:\/\/github.com\/utkuozbulak\/pytorch-cnn-visualizations."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/2\/204\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:20:53Z","timestamp":1760160053000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/2\/204"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,7]]},"references-count":53,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["e23020204"],"URL":"https:\/\/doi.org\/10.3390\/e23020204","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,7]]}}}