{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T01:21:10Z","timestamp":1782436870734,"version":"3.54.5"},"reference-count":42,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2021,9,19]],"date-time":"2021-09-19T00:00:00Z","timestamp":1632009600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>In this paper, we present the potential of Explainable Artificial Intelligence methods for decision support in medical image analysis scenarios. Using three types of explainable methods applied to the same medical image data set, we aimed to improve the comprehensibility of the decisions provided by the Convolutional Neural Network (CNN). In vivo gastral images obtained by a video capsule endoscopy (VCE) were the subject of visual explanations, with the goal of increasing health professionals\u2019 trust in black-box predictions. We implemented two post hoc interpretable machine learning methods, called Local Interpretable Model-Agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), and an alternative explanation approach, the Contextual Importance and Utility (CIU) method. The produced explanations were assessed by human evaluation. We conducted three user studies based on explanations provided by LIME, SHAP and CIU. Users from different non-medical backgrounds carried out a series of tests in a web-based survey setting and stated their experience and understanding of the given explanations. Three user groups (n = 20, 20, 20) with three distinct forms of explanations were quantitatively analyzed. We found that, as hypothesized, the CIU-explainable method performed better than both LIME and SHAP methods in terms of improving support for human decision-making and being more transparent and thus understandable to users. Additionally, CIU outperformed LIME and SHAP by generating explanations more rapidly. Our findings suggest that there are notable differences in human decision-making between various explanation support settings. In line with that, we present three potential explainable methods that, with future improvements in implementation, can be generalized to different medical data sets and can provide effective decision support to medical experts.<\/jats:p>","DOI":"10.3390\/make3030037","type":"journal-article","created":{"date-parts":[[2021,9,21]],"date-time":"2021-09-21T22:35:20Z","timestamp":1632263720000},"page":"740-770","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":143,"title":["Explainable Artificial Intelligence for Human Decision Support System in the Medical Domain"],"prefix":"10.3390","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5926-6151","authenticated-orcid":false,"given":"Samanta","family":"Knapi\u010d","sequence":"first","affiliation":[{"name":"Department of Computer Science, Aalto University, Konemiehentie 2, 02150 Espoo, Finland"},{"name":"Department of Computing Science, Ume\u00e5 University, 90187 Ume\u00e5, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avleen","family":"Malhi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalto University, Konemiehentie 2, 02150 Espoo, Finland"},{"name":"Department of Computing and Informatics, Bournemouth University, Poole BH12 5BB, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohit","family":"Saluja","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalto University, Konemiehentie 2, 02150 Espoo, Finland"},{"name":"Department of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 11428 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kary","family":"Fr\u00e4mling","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalto University, Konemiehentie 2, 02150 Espoo, Finland"},{"name":"Department of Computing Science, Ume\u00e5 University, 90187 Ume\u00e5, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Meske, C., and Bunde, E. (2020, January 19\u201324). Transparency and Trust in Human-AI-Interaction: The Role of Model-Agnostic Explanations in Computer Vision-Based Decision Support. Proceedings of the International Conference on Human-Computer Interaction, Germany, Copenhagen, Denmark.","DOI":"10.1007\/978-3-030-50334-5_4"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Coelho, P., Pereira, A., Salgado, M., and Cunha, A. (2018, January 27\u201329). A deep learning approach for red lesions detection in video capsule endoscopies. Proceedings of the International Conference Image Analysis and Recognition, Waterloo, ON, Canada.","DOI":"10.1007\/978-3-319-93000-8_63"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Malhi, A., Kampik, T., Pannu, H., Madhikermi, M., and Fr\u00e4mling, K. (2019, January 13\u201314). Explaining Machine Learning-Based Classifications of In-Vivo Gastral Images. Proceedings of the International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, Montreal, QC, Canada.","DOI":"10.1109\/DICTA47822.2019.8945986"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Malhi, A., Knapi\u010d, S., and Fr\u00e4mling, K. (2020, January 9\u201313). Explainable Agents for Less Bias in Human-Agent Decision Making. Proceedings of the International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, Auckland, New Zealand.","DOI":"10.1007\/978-3-030-51924-7_8"},{"key":"ref_5","unstructured":"Anjomshoae, S., Kampik, T., and Fr\u00e4mling, K. (2020, January 1). Py-CIU: A Python Library for Explaining Machine Learning Predictions Using Contextual Importance and Utility. Proceedings of the IJCAI-PRICAI 2020 Workshop on Explainable Artificial Intelligence (XAI), Yokohama, Japan."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_7","first-page":"93","article-title":"A survey of methods for explaining black box models","volume":"51","author":"Guidotti","year":"2018","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_8","unstructured":"Anjomshoae, S., Najjar, A., Calvaresi, D., and Fr\u00e4mling, K. (2019, January 13\u201317). Explainable agents and robots: Results from a systematic literature review. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, International Foundation for Autonomous Agents and Multiagent Systems, Montreal, QC, Canada."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.inffus.2021.05.009","article-title":"Notions of explainability and evaluation approaches for explainable artificial intelligence","volume":"76","author":"Vilone","year":"2021","journal-title":"Inf. Fusion"},{"key":"ref_10","unstructured":"Fr\u00e4mling, K., and Graillot, D. (1995, January 9\u201313). Extracting Explanations from Neural Networks. Proceedings of the ICANN, Citeseer, Paris, France."},{"key":"ref_11","unstructured":"Fr\u00e4mling, K. (1996). Mod\u00e9lisation et Apprentissage des Pr\u00e9f\u00e9rences par R\u00e9seaux de Neurones Pour L\u2019aide \u00e0 la D\u00e9cision Multicrit\u00e8re. [Ph.D. Thesis, INSA de Lyon]."},{"key":"ref_12","unstructured":"Lundberg, S. (2019, June 04). SHAP Python Package. Available online: https:\/\/github.com\/slundberg\/shap."},{"key":"ref_13","unstructured":"(2019, June 04). ELI5. Available online: https:\/\/github.com\/TeamHG-Memex\/eli5."},{"key":"ref_14","unstructured":"(2019, June 04). Skater. Available online: https:\/\/github.com\/oracle\/Skater."},{"key":"ref_15","unstructured":"Xie, N., Ras, G., van Gerven, M., and Doran, D. (2020). Explainable deep learning: A field guide for the uninitiated. arXiv."},{"key":"ref_16","unstructured":"Samek, W., Wiegand, T., and M\u00fcller, K.R. (2017). Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/MCG.2018.042731661","article-title":"Visual analytics for explainable deep learning","volume":"38","author":"Choo","year":"2018","journal-title":"IEEE Comput. Graph. Appl."},{"key":"ref_18","first-page":"44","article-title":"DARPA\u2019s explainable artificial intelligence (XAI) program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Mag."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4613","DOI":"10.1073\/pnas.1716999115","article-title":"An explainable deep machine vision framework for plant stress phenotyping","volume":"115","author":"Ghosal","year":"2018","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hase, P., and Bansal, M. (2020). Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.491"},{"key":"ref_21","unstructured":"Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv."},{"key":"ref_22","unstructured":"Holzinger, A., Malle, B., Kieseberg, P., Roth, P.M., M\u00fcller, H., Reihs, R., and Zatloukal, K. (2017). Towards the augmented pathologist: Challenges of explainable-ai in digital pathology. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.1002\/mp.13264","article-title":"Deep learning in medical imaging and radiation therapy","volume":"46","author":"Sahiner","year":"2019","journal-title":"Med. Phys."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12911-020-01332-6","article-title":"Explainability for artificial intelligence in healthcare: A multidisciplinary perspective","volume":"20","author":"Amann","year":"2020","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wang, S.H., Zhang, Y., Cheng, X., Zhang, X., and Zhang, Y.D. (2021). PSSPNN: PatchShuffle Stochastic Pooling Neural Network for an explainable diagnosis of COVID-19 with multiple-way data augmentation. Comput. Math. Methods Med.","DOI":"10.1155\/2021\/6633755"},{"key":"ref_27","first-page":"1037","article-title":"ANC: Attention network for COVID-19 explainable diagnosis based on convolutional block attention module","volume":"127","author":"Zhang","year":"2021","journal-title":"CMES-Comput. Model. Eng. Sci."},{"key":"ref_28","first-page":"3152676","article-title":"The eu general data protection regulation (gdpr)","volume":"Volume 10","author":"Voigt","year":"2017","journal-title":"A Practical Guide"},{"key":"ref_29","unstructured":"Egger, J., Gsaxner, C., Pepe, A., and Li, J. (2020). Medical Deep Learning\u2014A systematic Meta-Review. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"42200","DOI":"10.1109\/ACCESS.2020.2976199","article-title":"Explainable machine learning for scientific insights and discoveries","volume":"8","author":"Roscher","year":"2020","journal-title":"IEEE Access"},{"key":"ref_31","unstructured":"Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23.","DOI":"10.3390\/e23010018"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Do\u0161ilovi\u0107, F.K., Br\u010di\u0107, M., and Hlupi\u0107, N. (2018, January 21\u201325). Explainable artificial intelligence: A survey. Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia.","DOI":"10.23919\/MIPRO.2018.8400040"},{"key":"ref_34","unstructured":"Lundberg, S., and Lee, S.I. (2017). A unified approach to interpreting model predictions. arXiv."},{"key":"ref_35","unstructured":"Pedersen, T.L. (2020, September 20). LIME R Package. Available online: https:\/\/github.com\/thomasp85\/lime."},{"key":"ref_36","unstructured":"Ribeiro, M.T.C. (2020, September 10). LIME Python Package. Available online: https:\/\/github.com\/marcotcr\/lime."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Shapley, L. (1953). A Value for N-Person Games. Contributions to the Theory of Games (Vol. 2), Princeton University Press.","DOI":"10.1515\/9781400881970-018"},{"key":"ref_38","unstructured":"Fr\u00e4mling, K. (2020). Explainable AI without Interpretable Model. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Fr\u00e4mling, K., Westberg, M., Jullum, M., Madhikermi, M., and Malhi, A. (2021, January 3\u20137). Comparison of Contextual Importance and Utility with LIME and Shapley Values. Proceedings of the International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, London, UK (fully online).","DOI":"10.1007\/978-3-030-82017-6_3"},{"key":"ref_41","unstructured":"Molnar, C. (2020, September 12). Interpretable Machine Learning. Available online: https:\/\/christophm.github.io\/interpretable-ml-book\/."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Fr\u00e4mling, K., Knapi\u010d, S., and Malhi, A. (2021, January 3\u20137). ciu. image: An R Package for Explaining Image Classification with Contextual Importance and Utility. Proceedings of the International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, London, UK (fully online).","DOI":"10.1007\/978-3-030-82017-6_4"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/3\/37\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:02:14Z","timestamp":1760166134000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/3\/3\/37"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,19]]},"references-count":42,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["make3030037"],"URL":"https:\/\/doi.org\/10.3390\/make3030037","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,19]]}}}