{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T18:37:35Z","timestamp":1786214255883,"version":"3.56.0"},"reference-count":119,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T00:00:00Z","timestamp":1672876800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NATIONAL NATURAL SCIENCE FOUNDATION","award":["82260360"],"award-info":[{"award-number":["82260360"]}]},{"name":"NATIONAL NATURAL SCIENCE FOUNDATION","award":["QN2021033002L"],"award-info":[{"award-number":["QN2021033002L"]}]},{"name":"FOREIGN YOUNG TALENTS PROGRAM","award":["82260360"],"award-info":[{"award-number":["82260360"]}]},{"name":"FOREIGN YOUNG TALENTS PROGRAM","award":["QN2021033002L"],"award-info":[{"award-number":["QN2021033002L"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Artificial intelligence (AI) with deep learning models has been widely applied in numerous domains, including medical imaging and healthcare tasks. In the medical field, any judgment or decision is fraught with risk. A doctor will carefully judge whether a patient is sick before forming a reasonable explanation based on the patient\u2019s symptoms and\/or an examination. Therefore, to be a viable and accepted tool, AI needs to mimic human judgment and interpretation skills. Specifically, explainable AI (XAI) aims to explain the information behind the black-box model of deep learning that reveals how the decisions are made. This paper provides a survey of the most recent XAI techniques used in healthcare and related medical imaging applications. We summarize and categorize the XAI types, and highlight the algorithms used to increase interpretability in medical imaging topics. In addition, we focus on the challenging XAI problems in medical applications and provide guidelines to develop better interpretations of deep learning models using XAI concepts in medical image and text analysis. Furthermore, this survey provides future directions to guide developers and researchers for future prospective investigations on clinical topics, particularly on applications with medical imaging.<\/jats:p>","DOI":"10.3390\/s23020634","type":"journal-article","created":{"date-parts":[[2023,1,6]],"date-time":"2023-01-06T03:19:54Z","timestamp":1672975194000},"page":"634","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":544,"title":["Survey of Explainable AI Techniques in Healthcare"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3402-9576","authenticated-orcid":false,"given":"Ahmad","family":"Chaddad","sequence":"first","affiliation":[{"name":"School of Artificial Intelligence, Guilin University of Electronic Technology, Jinji Road, Guilin 541004, China"},{"name":"The Laboratory for Imagery Vision and Artificial Intelligence, Ecole de Technologie Superieure, 1100 Rue Notre Dame O, Montreal, QC H3C 1K3, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jihao","family":"Peng","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Guilin University of Electronic Technology, Jinji Road, Guilin 541004, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jian","family":"Xu","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Guilin University of Electronic Technology, Jinji Road, Guilin 541004, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1474-2772","authenticated-orcid":false,"given":"Ahmed","family":"Bouridane","sequence":"additional","affiliation":[{"name":"Centre for Data Analytics and Cybersecurity, University of Sharjah, Sharjah 27272, United Arab Emirates"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"153316","DOI":"10.1109\/ACCESS.2021.3127881","article-title":"A systematic review of human-computer interaction and explainable artificial intelligence in healthcare with artificial intelligence techniques","volume":"9","author":"Nazar","year":"2021","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1607","DOI":"10.1007\/s13347-021-00477-0","article-title":"Transparency and the black box problem: Why we do not trust AI","volume":"34","year":"2021","journal-title":"Philos. Technol."},{"key":"ref_3","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_4","first-page":"44","article-title":"DARPA\u2019s explainable artificial intelligence (XAI) program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Mag."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.neunet.2020.07.010","article-title":"Towards explainable deep neural networks (xDNN)","volume":"130","author":"Angelov","year":"2020","journal-title":"Neural Netw."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Arrieta","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","article-title":"A survey on explainable artificial intelligence (xai): Toward medical xai","volume":"32","author":"Tjoa","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nat. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"eaay7120","DOI":"10.1126\/scirobotics.aay7120","article-title":"XAI\u2014Explainable artificial intelligence","volume":"4","author":"Gunning","year":"2019","journal-title":"Sci. Robot."},{"key":"ref_10","unstructured":"Yang, X., He, X., Zhao, J., Zhang, Y., Zhang, S., and Xie, P. (2020). COVID-CT-dataset: A CT scan dataset about COVID-19. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1038\/s41592-018-0261-2","article-title":"U-Net: Deep learning for cell counting, detection, and morphometry","volume":"16","author":"Falk","year":"2019","journal-title":"Nat. Methods"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"97","DOI":"10.9785\/cri-2019-200402","article-title":"The EU approach to ethics guidelines for trustworthy artificial intelligence","volume":"20","author":"Smuha","year":"2019","journal-title":"Comput. Law Rev. Int."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bai, T., Zhao, J., Zhu, J., Han, S., Chen, J., Li, B., and Kot, A. (2021, January 19\u201322). Ai-gan: Attack-inspired generation of adversarial examples. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.","DOI":"10.1109\/ICIP42928.2021.9506278"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., and Swami, A. (2016, January 21\u201324). The limitations of deep learning in adversarial settings. Proceedings of the 2016 IEEE European symposium on security and privacy (EuroS&P), Saarbr\u00fccken, Germany.","DOI":"10.1109\/EuroSP.2016.36"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1007\/s00146-020-01085-w","article-title":"Artificial intelligence in medicine and the disclosure of risks","volume":"36","author":"Kiener","year":"2021","journal-title":"AI Soc."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Vigano, L., and Magazzeni, D. (2020, January 7\u201311). Explainable security. Proceedings of the 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), Genoa, Italy.","DOI":"10.1109\/EuroSPW51379.2020.00045"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kuppa, A., and Le-Khac, N.A. (2020, January 19\u201324). Black Box Attacks on Explainable Artificial Intelligence(XAI) methods in Cyber Security. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9206780"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Trocin, C., Mikalef, P., Papamitsiou, Z., and Conboy, K. (2021). Responsible AI for digital health: A synthesis and a research agenda. Inf. Syst. Front., 1\u201319.","DOI":"10.1007\/s10796-021-10146-4"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e200267","DOI":"10.1148\/ryai.2021200267","article-title":"Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging","volume":"3","author":"Arun","year":"2021","journal-title":"Radiol. Artif. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1007\/s00146-020-01019-6","article-title":"Clinical AI: Opacity, accountability, responsibility and liability","volume":"36","author":"Smith","year":"2021","journal-title":"AI Soc."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/s43681-020-00009-0","article-title":"Responsible AI and moral responsibility: A common appreciation","volume":"1","author":"Tigard","year":"2021","journal-title":"AI Ethics"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A., and Ferrer, C.C. (2021, January 20\u201325). Casual conversations: A dataset for measuring fairness in ai. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPRW53098.2021.00258"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1038\/538020a","article-title":"Can we open the black box of AI?","volume":"538","author":"Castelvecchi","year":"2016","journal-title":"Nat. News"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi","year":"2021","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery","volume":"16","author":"Lipton","year":"2018","journal-title":"Queue"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1145\/3359786","article-title":"Techniques for Interpretable Machine Learning","volume":"63","author":"Du","year":"2019","journal-title":"Commun. ACM"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.inffus.2021.07.016","article-title":"Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond","volume":"77","author":"Yang","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_28","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv."},{"key":"ref_29","unstructured":"Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2014). Striving for simplicity: The all convolutional net. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014, January 6\u201312). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27\u201330). Learning deep features for discriminative localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.319"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should i trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Huang, Q., Yamada, M., Tian, Y., Singh, D., and Chang, Y. (2022). Graphlime: Local interpretable model explanations for graph neural networks. IEEE Trans. Knowl. Data Eng.","DOI":"10.1109\/TKDE.2022.3187455"},{"key":"ref_36","first-page":"4768","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_37","unstructured":"Jetley, S., Lord, N.A., Lee, N., and Torr, P.H.S. (2018). Learn To Pay Attention. arXiv."},{"key":"ref_38","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). This Looks Like That: Deep Learning for Interpretable Image Recognition. Proceedings of the Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Hoffer, E., and Ailon, N. (2015, January 12\u201314). Deep metric learning using triplet network. Proceedings of the International Workshop on Similarity-Based Pattern Recognition, Copenhagen, Denmark.","DOI":"10.1007\/978-3-319-24261-3_7"},{"key":"ref_40","first-page":"2668","article-title":"Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)","volume":"Volume 80","author":"Dy","year":"2018","journal-title":"Proceedings of the 35th International Conference on Machine Learning"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7\u201312). Show and Tell: A Neural Image Caption Generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1038\/s41591-020-01192-7","article-title":"An algorithmic approach to reducing unexplained pain disparities in underserved populations","volume":"27","author":"Pierson","year":"2021","journal-title":"Nat. Med."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Born, J., Wiedemann, N., Cossio, M., Buhre, C., Br\u00e4ndle, G., Leidermann, K., Goulet, J., Aujayeb, A., Moor, M., and Rieck, B. (2021). Accelerating detection of lung pathologies with explainable ultrasound image analysis. Appl. Sci., 11.","DOI":"10.3390\/app11020672"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"101908","DOI":"10.1016\/j.media.2020.101908","article-title":"An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization","volume":"68","author":"Shen","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"104425","DOI":"10.1016\/j.compbiomed.2021.104425","article-title":"Classification of COVID-19 chest X-ray and CT images using a type of dynamic CNN modification method","volume":"134","author":"Jia","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"2775","DOI":"10.1109\/TCBB.2021.3065361","article-title":"Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images","volume":"18","author":"Song","year":"2021","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1016\/j.inffus.2020.10.004","article-title":"COVID-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network","volume":"67","author":"Wang","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Fan, Z., Gong, P., Tang, S., Lee, C.U., Zhang, X., Song, P., Chen, S., and Li, H. (2022). Joint localization and classification of breast tumors on ultrasound images using a novel auxiliary attention-based framework. arXiv.","DOI":"10.1016\/j.media.2023.102960"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"107613","DOI":"10.1016\/j.patcog.2020.107613","article-title":"Automatically discriminating and localizing COVID-19 from community-acquired pneumonia on chest X-rays","volume":"110","author":"Wang","year":"2021","journal-title":"Pattern Recognit."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-022-06726-2","article-title":"Artificial intelligence enabled automated diagnosis and grading of ulcerative colitis endoscopy images","volume":"12","author":"Sutton","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1016\/S1470-2045(20)30535-0","article-title":"Deep learning model for the prediction of microsatellite instability in colorectal cancer: A diagnostic study","volume":"22","author":"Yamashita","year":"2021","journal-title":"Lancet Oncol."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"3113","DOI":"10.1109\/TIP.2021.3058783","article-title":"Jcs: An explainable covid-19 diagnosis system by joint classification and segmentation","volume":"30","author":"Wu","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1572","DOI":"10.1002\/int.22686","article-title":"NAGNN: Classification of COVID-19 based on neighboring aware representation from deep graph neural network","volume":"37","author":"Lu","year":"2022","journal-title":"Int. J. Intell. Syst."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"30615","DOI":"10.1007\/s11042-022-12156-z","article-title":"COVID-cxnet: Detecting COVID-19 in frontal chest X-ray images using deep learning","volume":"81","author":"Haghanifar","year":"2022","journal-title":"Multimed. Tools Appl."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"2689","DOI":"10.1007\/s10489-020-01900-3","article-title":"Automated diagnosis of COVID-19 with limited posteroanterior chest X-ray images using fine-tuned deep neural networks","volume":"51","author":"Punn","year":"2021","journal-title":"Appl. Intell."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"101846","DOI":"10.1016\/j.media.2020.101846","article-title":"Triple attention learning for classification of 14 thoracic diseases using chest radiography","volume":"67","author":"Wang","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"3507","DOI":"10.1109\/JBHI.2021.3059453","article-title":"Multimodal spatial attention module for targeting multimodal PET-CT lung tumor segmentation","volume":"25","author":"Fu","year":"2021","journal-title":"IEEE J. Biomed. Health Inf."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"104815","DOI":"10.1016\/j.compbiomed.2021.104815","article-title":"Focus U-Net: A novel dual attention-gated CNN for polyp segmentation during colonoscopy","volume":"137","author":"Yeung","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Hu, B., Vasu, B., and Hoogs, A. (2022, January 4\u20138). X-MIR: EXplainable Medical Image Retrieval. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.","DOI":"10.1109\/WACV51458.2022.00161"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1038\/s41598-021-04608-7","article-title":"An explainable machine learning framework for lung cancer hospital length of stay prediction","volume":"12","author":"Alsinglawi","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"1170","DOI":"10.1038\/s41598-022-05112-2","article-title":"An explainable machine learning-based clinical decision support system for prediction of gestational diabetes mellitus","volume":"12","author":"Du","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Severn, C., Suresh, K., G\u00f6rg, C., Choi, Y.S., Jain, R., and Ghosh, D. (2022). A Pipeline for the Implementation and Visualization of Explainable Machine Learning for Medical Imaging Using Radiomics Features. Sensors, 22.","DOI":"10.3390\/s22145205"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Le, N.Q.K., Kha, Q.H., Nguyen, V.H., Chen, Y.C., Cheng, S.J., and Chen, C.Y. (2021). Machine learning-based radiomics signatures for EGFR and KRAS mutations prediction in non-small-cell lung cancer. Int. J. Mol. Sci., 22.","DOI":"10.3390\/ijms22179254"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"6968","DOI":"10.1038\/s41598-021-86327-7","article-title":"Explainable machine learning can outperform Cox regression predictions and provide insights in breast cancer survival","volume":"11","author":"Hendriks","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Abeyagunasekera, S.H.P., Perera, Y., Chamara, K., Kaushalya, U., Sumathipala, P., and Senaweera, O. (2022, January 7\u20139). LISA: Enhance the explainability of medical images unifying current XAI techniques. Proceedings of the 2022 IEEE 7th International Conference for Convergence in Technology (I2CT), Mumbai, India.","DOI":"10.1109\/I2CT54291.2022.9824840"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Duell, J., Fan, X., Burnett, B., Aarts, G., and Zhou, S.M. (2021, January 27\u201330). A Comparison of Explanations Given by Explainable Artificial Intelligence Methods on Analysing Electronic Health Records. Proceedings of the 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), Athens, Greece.","DOI":"10.1109\/BHI50953.2021.9508618"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"3786","DOI":"10.1109\/TNNLS.2021.3099165","article-title":"Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation with Alternate Learning","volume":"32","author":"Liu","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","year":"2021","journal-title":"Lancet Digit. Health"},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Wu, J., and Mooney, R.J. (2018). Faithful Multimodal Explanation for Visual Question Answering. arXiv.","DOI":"10.18653\/v1\/W19-4812"},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_71","unstructured":"Das, A., and Rad, P. (2020). Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv."},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Chen, P., Sapkota, M., and Yang, L. (2017, January 10\u201314). Tandemnet: Distilling knowledge from medical images using diagnostic reports as optional semantic references. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada.","DOI":"10.1007\/978-3-319-66179-7_37"},{"key":"ref_73","first-page":"3530","article-title":"Deep Learning for Case-Based Reasoning Through Prototypes: A Neural Network That Explains Its Predictions","volume":"32","author":"Li","year":"2018","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/j.patcog.2016.11.008","article-title":"Explaining nonlinear classification decisions with deep taylor decomposition","volume":"65","author":"Montavon","year":"2017","journal-title":"Pattern Recognit."},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"102239","DOI":"10.1016\/j.displa.2022.102239","article-title":"A review of visualisation-as-explanation techniques for convolutional neural networks and their evaluation","volume":"73","author":"Mohamed","year":"2022","journal-title":"Displays"},{"key":"ref_76","unstructured":"Soares, E., Angelov, P., Biaso, S., Froes, M.H., and Abe, D.K. (2020). SARS-CoV-2 CT-scan dataset: A large dataset of real patients CT scans for SARS-CoV-2 identification. MedRxiv."},{"key":"ref_77","unstructured":"Jiang, C., Chen, Y., Chang, J., Feng, M., Wang, R., and Yao, J. (2021). Fusion of medical imaging and electronic health records with attention and multi-head machanisms. arXiv."},{"key":"ref_78","first-page":"18754","article-title":"Dual Decomposition of Convex Optimization Layers for Consistent Attention in Medical Images","volume":"Volume 162","author":"Chaudhuri","year":"2022","journal-title":"Proceedings of the 39th International Conference on Machine Learning"},{"key":"ref_79","unstructured":"Shapley, L.S. (2016). 17. A value for n-person games. Contributions to the Theory of Games (AM-28), Volume II, Princeton University Press."},{"key":"ref_80","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Learning Important Features Through Propagating Activation Differences. arXiv."},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Liu, F., Ge, S., and Wu, X. (2022). Competence-based multimodal curriculum learning for medical report generation. arXiv.","DOI":"10.18653\/v1\/2021.acl-long.234"},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Malhi, A., Kampik, T., Pannu, H., Madhikermi, M., and Fr\u00e4mling, K. (2019, January 2\u20134). Explaining machine learning-based classifications of in-vivo gastral images. Proceedings of the 2019 Digital Image Computing: Techniques and Applications (DICTA), Perth, Australia.","DOI":"10.1109\/DICTA47822.2019.8945986"},{"key":"ref_84","doi-asserted-by":"crossref","unstructured":"Ye, Q., Xia, J., and Yang, G. (2021, January 7\u20139). Explainable AI for COVID-19 CT classifiers: An initial comparison study. Proceedings of the 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), Aveiro, Portugal.","DOI":"10.1109\/CBMS52027.2021.00103"},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Serrano, S., and Smith, N.A. (2019). Is attention interpretable?. arXiv.","DOI":"10.18653\/v1\/P19-1282"},{"key":"ref_86","doi-asserted-by":"crossref","first-page":"374","DOI":"10.3389\/fonc.2019.00374","article-title":"Radiomics in glioblastoma: Current status and challenges facing clinical implementation","volume":"9","author":"Chaddad","year":"2019","journal-title":"Front. Oncol."},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Chaddad, A., Kucharczyk, M.J., Cheddad, A., Clarke, S.E., Hassan, L., Ding, S., Rathore, S., Zhang, M., Katib, Y., and Bahoric, B. (2021). Magnetic resonance imaging based radiomic models of prostate cancer: A narrative review. Cancers, 13.","DOI":"10.3390\/cancers13030552"},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"97242","DOI":"10.1109\/ACCESS.2019.2930238","article-title":"Deep radiomic analysis based on modeling information flow in convolutional neural networks","volume":"7","author":"Chaddad","year":"2019","journal-title":"IEEE Access"},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1038\/s41416-021-01387-w","article-title":"Radiomics and radiogenomics in gliomas: A contemporary update","volume":"125","author":"Singh","year":"2021","journal-title":"Br. J. Cancer"},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/TNNLS.2021.3119071","article-title":"Deep radiomic analysis for predicting coronavirus disease 2019 in computerized tomography and X-ray images","volume":"33","author":"Chaddad","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_91","doi-asserted-by":"crossref","unstructured":"Gupta, S., and Gupta, M. (2021, January 13\u201315). Deep Learning for Brain Tumor Segmentation using Magnetic Resonance Images. Proceedings of the 2021 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), Melbourne, Australia.","DOI":"10.1109\/CIBCB49929.2021.9562890"},{"key":"ref_92","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1016\/j.neucom.2020.10.117","article-title":"Deep radiomic signature with immune cell markers predicts the survival of glioma patients","volume":"469","author":"Chaddad","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Chaddad, A., Zhang, M., Desrosiers, C., and Niazi, T. (2019, January 13). Deep radiomic features from MRI scans predict survival outcome of recurrent glioblastoma. Proceedings of the International Workshop on Radiomics and Radiogenomics in Neuro-Oncology, Shenzhen, China.","DOI":"10.1007\/978-3-030-40124-5_4"},{"key":"ref_94","doi-asserted-by":"crossref","unstructured":"Moridian, P., Ghassemi, N., Jafari, M., Salloum-Asfar, S., Sadeghi, D., Khodatars, M., Shoeibi, A., Khosravi, A., Ling, S.H., and Subasi, A. (2022). Automatic Autism Spectrum Disorder Detection Using Artificial Intelligence Methods with MRI Neuroimaging: A Review. arXiv.","DOI":"10.3389\/fnmol.2022.999605"},{"key":"ref_95","doi-asserted-by":"crossref","first-page":"1296","DOI":"10.1007\/s11547-021-01389-x","article-title":"A deep look into radiomics","volume":"126","author":"Scapicchio","year":"2021","journal-title":"Radiol. Med."},{"key":"ref_96","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/S2468-1253(20)30290-9","article-title":"Personalised versus standard dosimetry approach of selective internal radiation therapy in patients with locally advanced hepatocellular carcinoma (DOSISPHERE-01): A randomised, multicentre, open-label phase 2 trial","volume":"6","author":"Garin","year":"2021","journal-title":"Lancet Gastroenterol. Hepatol."},{"key":"ref_97","doi-asserted-by":"crossref","first-page":"103581","DOI":"10.1016\/j.isci.2021.103581","article-title":"CX-ToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models","volume":"25","author":"Akula","year":"2022","journal-title":"iScience"},{"key":"ref_98","doi-asserted-by":"crossref","unstructured":"Ehsan, U., Liao, Q.V., Muller, M., Riedl, M.O., and Weisz, J.D. (2021, January 8\u201313). Expanding explainability: Towards social transparency in ai systems. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411764.3445188"},{"key":"ref_99","doi-asserted-by":"crossref","unstructured":"Kumarakulasinghe, N.B., Blomberg, T., Liu, J., Leao, A.S., and Papapetrou, P. (2020, January 28\u201330). Evaluating local interpretable model-agnostic explanations on clinical machine learning classification models. Proceedings of the 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS), Rochester, MN, USA.","DOI":"10.1109\/CBMS49503.2020.00009"},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1016\/j.future.2022.03.009","article-title":"The explainability paradox: Challenges for xAI in digital pathology","volume":"133","author":"Evans","year":"2022","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_101","doi-asserted-by":"crossref","first-page":"105111","DOI":"10.1016\/j.compbiomed.2021.105111","article-title":"Transparency of deep neural networks for medical image analysis: A review of interpretability methods","volume":"140","author":"Salahuddin","year":"2022","journal-title":"Comput. Biol. Med."},{"key":"ref_102","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1109\/THMS.2022.3144956","article-title":"A Review on Human\u2013Machine Trust Evaluation: Human-Centric and Machine-Centric Perspectives","volume":"52","author":"Gebru","year":"2022","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"ref_103","unstructured":"Adebayo, J., Muelly, M., Abelson, H., and Kim, B. (2021, January 3\u20137). Post hoc explanations may be ineffective for detecting unknown spurious correlation. Proceedings of the International Conference on Learning Representations, Virtual Event."},{"key":"ref_104","doi-asserted-by":"crossref","unstructured":"Alqaraawi, A., Schuessler, M., Wei\u00df, P., Costanza, E., and Berthouze, N. (2020, January 17\u201320). Evaluating saliency map explanations for convolutional neural networks: A user study. Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari, Italy.","DOI":"10.1145\/3377325.3377519"},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"11974","DOI":"10.1109\/ACCESS.2021.3051315","article-title":"A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence","volume":"9","author":"Stepin","year":"2021","journal-title":"IEEE Access"},{"key":"ref_106","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1038\/s41746-020-0221-y","article-title":"An overview of clinical decision support systems: Benefits, risks, and strategies for success","volume":"3","author":"Sutton","year":"2020","journal-title":"NPJ Digit. Med."},{"key":"ref_107","doi-asserted-by":"crossref","unstructured":"Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., and Gebru, T. (2019, January 29\u201331). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA.","DOI":"10.1145\/3287560.3287596"},{"key":"ref_108","doi-asserted-by":"crossref","first-page":"6:1","DOI":"10.1147\/JRD.2019.2942288","article-title":"FactSheets: Increasing trust in AI services through supplier\u2019s declarations of conformity","volume":"63","author":"Arnold","year":"2019","journal-title":"IBM J. Res. Dev."},{"key":"ref_109","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1145\/3458723","article-title":"Datasheets for datasets","volume":"64","author":"Gebru","year":"2021","journal-title":"Commun. ACM"},{"key":"ref_110","doi-asserted-by":"crossref","first-page":"102470","DOI":"10.1016\/j.media.2022.102470","article-title":"Explainable artificial intelligence (XAI) in deep learning-based medical image analysis","volume":"79","author":"Kuijf","year":"2022","journal-title":"Med. Image Anal."},{"key":"ref_111","doi-asserted-by":"crossref","unstructured":"Lopes, P., Silva, E., Braga, C., Oliveira, T., and Rosado, L. (2022). XAI Systems Evaluation: A Review of Human and Computer-Centred Methods. Appl. Sci., 12.","DOI":"10.3390\/app12199423"},{"key":"ref_112","doi-asserted-by":"crossref","unstructured":"Lin, Y.S., Lee, W.C., and Celik, Z.B. (2020). What do you see? Evaluation of explainable artificial intelligence (XAI) interpretability through neural backdoors. arXiv.","DOI":"10.1145\/3447548.3467213"},{"key":"ref_113","unstructured":"Nguyen, H.T.T., Cao, H.Q., Nguyen, K.V.T., and Pham, N.D.K. (2021, January 6\u20137). Evaluation of Explainable Artificial Intelligence: SHAP, LIME, and CAM. Proceedings of the FPT AI Conference 2021, Ha Noi, Viet Nam."},{"key":"ref_114","doi-asserted-by":"crossref","unstructured":"Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schl\u00f6tterer, J., van Keulen, M., and Seifert, C. (2022). From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. arXiv.","DOI":"10.1145\/3583558"},{"key":"ref_115","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Xu, F., Zou, J., Petrosian, O.L., and Krinkin, K.V. (2021, January 16). XAI Evaluation: Evaluating Black-Box Model Explanations for Prediction. Proceedings of the 2021 II International Conference on Neural Networks and Neurotechnologies (NeuroNT), Saint Petersburg, Russia.","DOI":"10.1109\/NeuroNT53022.2021.9472817"},{"key":"ref_116","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Wu, Y.N., and Zhu, S.C. (2018, January 18\u201323). Interpretable Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00920"},{"key":"ref_117","unstructured":"Geiger, A., Wu, Z., Lu, H., Rozner, J., Kreiss, E., Icard, T., Goodman, N., and Potts, C. (2022, January 17\u201323). Inducing causal structure for interpretable neural networks. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA."},{"key":"ref_118","first-page":"9127","article-title":"ProtGNN: Towards Self-Explaining Graph Neural Networks","volume":"36","author":"Zhang","year":"2022","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_119","doi-asserted-by":"crossref","unstructured":"Chaddad, A., Li, J., Katib, Y., Kateb, R., Tanougast, C., Bouridane, A., and Abdulkadir, A. (2022). Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine. arXiv.","DOI":"10.1109\/JAS.2023.123123"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/2\/634\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:00:27Z","timestamp":1760119227000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/2\/634"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,5]]},"references-count":119,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["s23020634"],"URL":"https:\/\/doi.org\/10.3390\/s23020634","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,5]]}}}