{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T20:42:32Z","timestamp":1783197752676,"version":"3.54.6"},"reference-count":169,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,1,6]],"date-time":"2023-01-06T00:00:00Z","timestamp":1672963200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,6]],"date-time":"2023-01-06T00:00:00Z","timestamp":1672963200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["FKZ 01IS18056 B"],"award-info":[{"award-number":["FKZ 01IS18056 B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In the meantime, a wide variety of terminologies, motivations, approaches, and evaluation criteria have been developed within the research field of explainable artificial intelligence (XAI). With the amount of XAI methods vastly growing, a taxonomy of methods is needed by researchers as well as practitioners: To grasp the breadth of the topic, compare methods, and to select the right XAI method based on traits required by a specific use-case context. Many taxonomies for XAI methods of varying level of detail and depth can be found in the literature. While they often have a different focus, they also exhibit many points of overlap. This paper unifies these efforts and provides a complete taxonomy of XAI methods with respect to notions present in the current state of research. In a structured literature analysis and meta-study, we identified and reviewed more than 50 of the most cited and current surveys on XAI methods, metrics, and method traits. After summarizing them in a survey of surveys, we merge terminologies and concepts of the articles into a unified structured taxonomy. Single concepts therein are illustrated by more than 50 diverse selected example methods in total, which we categorize accordingly. The taxonomy may serve both beginners, researchers, and practitioners as a reference and wide-ranging overview of XAI method traits and aspects. Hence, it provides foundations for targeted, use-case-oriented, and context-sensitive future research.<\/jats:p>","DOI":"10.1007\/s10618-022-00867-8","type":"journal-article","created":{"date-parts":[[2023,1,6]],"date-time":"2023-01-06T17:02:56Z","timestamp":1673024576000},"page":"3043-3101","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":255,"title":["A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2690-2478","authenticated-orcid":false,"given":"Gesina","family":"Schwalbe","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9415-6254","authenticated-orcid":false,"given":"Bettina","family":"Finzel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,1,6]]},"reference":[{"key":"867_CR1","doi-asserted-by":"publisher","unstructured":"Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). In: IEEE Access, pp 52,138\u201352,160. https:\/\/doi.org\/10.1109\/ACCESS.2018.2870052","DOI":"10.1109\/ACCESS.2018.2870052"},{"key":"867_CR2","doi-asserted-by":"publisher","unstructured":"Alber M (2019) Software and application patterns for explanation methods. In: Explainable AI: interpreting, explaining and visualizing deep learning. Lecture notes in computer science. Springer, pp 399\u2013433. https:\/\/doi.org\/10.1007\/978-3-030-28954-6_22","DOI":"10.1007\/978-3-030-28954-6_22"},{"issue":"93","key":"867_CR3","first-page":"1","volume":"20","author":"M Alber","year":"2019","unstructured":"Alber M, Lapuschkin S, Seegerer P et al (2019) iNNvestigate neural networks. J Mach Learn Res 20(93):1\u20138","journal-title":"J Mach Learn Res"},{"key":"867_CR4","unstructured":"Allahyari H, Lavesson N (2011) User-oriented assessment of classification model understandability. In: 11th Scandinavian conference on artificial intelligence. IOS Press, http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:bth-7559"},{"issue":"3","key":"867_CR5","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1080\/00031305.1992.10475879","volume":"46","author":"NS Altman","year":"1992","unstructured":"Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175\u2013185. https:\/\/doi.org\/10.1080\/00031305.1992.10475879","journal-title":"Am Stat"},{"issue":"4","key":"867_CR6","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1609\/aimag.v35i4.2513","volume":"35","author":"S Amershi","year":"2014","unstructured":"Amershi S, Cakmak M, Knox WB et al (2014) Power to the people: the role of humans in interactive machine learning. AI Mag 35(4):105\u2013120. https:\/\/doi.org\/10.1609\/aimag.v35i4.2513","journal-title":"AI Mag"},{"key":"867_CR7","doi-asserted-by":"publisher","unstructured":"Ancona M, Ceolini E, \u00d6ztireli C, et\u00a0al (2019) Gradient-based attribution methods. In: Explainable AI: interpreting, explaining and visualizing deep learning. Lecture notes in computer science. Springer, pp 169\u2013191, https:\/\/doi.org\/10.1007\/978-3-030-28954-6_9","DOI":"10.1007\/978-3-030-28954-6_9"},{"key":"867_CR8","unstructured":"Anjomshoae S, Najjar A, Calvaresi D, et\u00a0al (2019) Explainable agents and robots: results from a systematic literature review. In: 18th international conference autonomous agents and multiagent systems (AAMAS 2019). International Foundation for Autonomous Agents and MultiAgent Systems, pp 1078\u20131088, http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:umu:diva-158024"},{"key":"867_CR9","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, Rodr\u00edguez ND, Ser JD et al (2020) Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fus. 58:82\u2013115. https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012","journal-title":"Inf. Fus."},{"key":"867_CR10","unstructured":"Artelt A, Hammer B (2019) On the computation of counterfactual explanations\u2014a survey. arXiv:1911.07749 [cs, stat]"},{"key":"867_CR11","unstructured":"Arya V, Bellamy RKE, Chen PY, et\u00a0al (2019) One explanation does not fit all: a toolkit and taxonomy of AI explainability techniques. CoRR arXiv:1909.03012"},{"key":"867_CR12","doi-asserted-by":"publisher","unstructured":"Augasta MG, Kathirvalavakumar T (2012) Rule extraction from neural networks\u2014a comparative study. In: Proceedings of the 2012 international conference pattern recognition, informatics and medical engineering, pp 404\u2013408. https:\/\/doi.org\/10.1109\/ICPRIME.2012.6208380,","DOI":"10.1109\/ICPRIME.2012.6208380"},{"issue":"7","key":"867_CR13","doi-asserted-by":"publisher","first-page":"e0130140","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach S, Binder A, Montavon G et al (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7):e0130140. https:\/\/doi.org\/10.1371\/journal.pone.0130140","journal-title":"PLoS ONE"},{"key":"867_CR14","first-page":"1803","volume":"11","author":"D Baehrens","year":"2010","unstructured":"Baehrens D, Schroeter T, Harmeling S et al (2010) How to explain individual classification decisions. J Mach Learn Res 11:1803\u20131831","journal-title":"J Mach Learn Res"},{"key":"867_CR15","unstructured":"Baniecki H, Biecek P (2020) The grammar of interactive explanatory model analysis. arXiv:2005.00497 [Cs Stat]"},{"key":"867_CR16","doi-asserted-by":"publisher","unstructured":"Bau D, Zhou B, Khosla A, et\u00a0al (2017) Network dissection: Quantifying interpretability of deep visual representations. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition. IEEE Computer Society, pp 3319\u20133327. https:\/\/doi.org\/10.1109\/CVPR.2017.354","DOI":"10.1109\/CVPR.2017.354"},{"key":"867_CR17","doi-asserted-by":"crossref","unstructured":"Belle V (2017) Logic meets probability: towards explainable ai systems for uncertain worlds. In: 26th international joint conference on artificial intelligence, pp 5116\u20135120","DOI":"10.24963\/ijcai.2017\/733"},{"key":"867_CR18","unstructured":"Benchekroun O, Rahimi A, Zhang Q, et\u00a0al (2020) The need for standardized explainability. arXiv:2010.11273 [Cs]"},{"key":"867_CR19","unstructured":"Biran O, Cotton CV (2017) Explanation and justification in machine learning: a survey. In: Proceedings of the IJCAI 2017 workshop explainable artificial intelligence (XAI)"},{"key":"867_CR20","unstructured":"Bodria F, Giannotti F, Guidotti R, et\u00a0al (2021) Benchmarking and survey of explanation methods for black box models. arXiv:2102.13076 [cs]"},{"key":"867_CR21","doi-asserted-by":"publisher","first-page":"75","DOI":"10.3389\/frai.2020.507973","volume":"3","author":"S Bruckert","year":"2020","unstructured":"Bruckert S, Finzel B, Schmid U (2020) The next generation of medical decision support: a roadmap toward transparent expert companions. Front Artif Intell 3:75","journal-title":"Front Artif Intell"},{"key":"867_CR22","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1613\/jair.1.12228","volume":"70","author":"N Burkart","year":"2021","unstructured":"Burkart N, Huber MF (2021) A survey on the explainability of supervised machine learning. J Artif Intell Res 70:245\u2013317. https:\/\/doi.org\/10.1613\/jair.1.12228","journal-title":"J Artif Intell Res"},{"key":"867_CR23","doi-asserted-by":"crossref","unstructured":"Byrne RMJ (2019) Counterfactuals in explainable artificial intelligence (XAI): evidence from human reasoning. In: Proceedings of the 2019 international joint conference artificial intelligence, pp 6276\u20136282. https:\/\/www.ijcai.org\/proceedings\/2019\/876","DOI":"10.24963\/ijcai.2019\/876"},{"issue":"1","key":"867_CR24","doi-asserted-by":"publisher","first-page":"7","DOI":"10.3233\/IA-190036","volume":"14","author":"R Calegari","year":"2020","unstructured":"Calegari R, Ciatto G, Omicini A (2020) On the integration of symbolic and sub-symbolic techniques for XAI: a survey. Intell Artif 14(1):7\u201332. https:\/\/doi.org\/10.3233\/IA-190036","journal-title":"Intell Artif"},{"issue":"8","key":"867_CR25","doi-asserted-by":"publisher","first-page":"832","DOI":"10.3390\/electronics8080832","volume":"8","author":"DV Carvalho","year":"2019","unstructured":"Carvalho DV, Pereira EM, Cardoso JS (2019) Machine learning interpretability: a survey on methods and metrics. Electronics 8(8):832. https:\/\/doi.org\/10.3390\/electronics8080832","journal-title":"Electronics"},{"key":"867_CR26","doi-asserted-by":"crossref","unstructured":"Chang CH, Tan S, Lengerich B, et\u00a0al (2020) How interpretable and trustworthy are GAMs? CoRR arXiv:2006.06466","DOI":"10.1145\/3447548.3467453"},{"issue":"3","key":"867_CR27","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1177\/1473871620904671","volume":"19","author":"A Chatzimparmpas","year":"2020","unstructured":"Chatzimparmpas A, Martins RM, Jusufi I et al (2020) A survey of surveys on the use of visualization for interpreting machine learning models. Inf Vis 19(3):207\u2013233. https:\/\/doi.org\/10.1177\/1473871620904671","journal-title":"Inf Vis"},{"key":"867_CR28","unstructured":"Chen C, Li O, Tao D et al (2019a) This looks like that: deep learning for interpretable image recognition. Adv Neural Inf Process Syst 32:8928\u20138939"},{"key":"867_CR29","doi-asserted-by":"publisher","unstructured":"Chen R, Chen H, Huang G, et\u00a0al (2019b) Explaining neural networks semantically and quantitatively. In: Proceedings of the 2019 IEEE\/CVF international conference on computer vision. IEEE, pp 9186\u20139195. https:\/\/doi.org\/10.1109\/ICCV.2019.00928","DOI":"10.1109\/ICCV.2019.00928"},{"key":"867_CR30","doi-asserted-by":"crossref","unstructured":"Chen Z, Bei Y, Rudin C (2020) Concept whitening for interpretable image recognition. CoRR arXiv:2002.01650","DOI":"10.1038\/s42256-020-00265-z"},{"key":"867_CR31","unstructured":"Choudhary P (2018) Interpreting predictive models with skater: unboxing model opacity. O\u2019Reilly Media https:\/\/www.oreilly.com\/content\/interpreting-predictive-models-with-skater-unboxing-model-opacity\/"},{"key":"867_CR32","unstructured":"Chromik M, Sch\u00fc\u00dfler M (2020) A taxonomy for human subject evaluation of black-box explanations in XAI. In: Proceedings of the workshop explainable smart systems for algorithmic transparency in emerging technologies, vol 2582. CEUR-WS.org, p\u00a07"},{"key":"867_CR33","unstructured":"Council AUPP (2017) Statement on algorithmic transparency and accountability. Commun ACM"},{"issue":"03","key":"867_CR34","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1142\/S0218213092000260","volume":"1","author":"MW Craven","year":"1992","unstructured":"Craven MW, Shavlik JW (1992) Visualizing learning and computation in artificial neural networks. Int J Artif Intell Tools 1(03):399\u2013425","journal-title":"Int J Artif Intell Tools"},{"key":"867_CR35","unstructured":"Craven MW, Shavlik JW (1995) Extracting tree-structured representations of trained networks. In: Advances in Neural Information Processing Systems 8, NIPS, Denver, CO, USA, November 27\u201330, 1995. MIT Press, pp 24\u201330, http:\/\/papers.nips.cc\/paper\/1152-extracting-tree-structured-representations-of-trained-networks"},{"key":"867_CR36","doi-asserted-by":"crossref","unstructured":"Cropper A, Dumancic S, Muggleton SH (2020) Turning 30: new ideas in inductive logic programming. CoRR arXiv:2002.11002","DOI":"10.24963\/ijcai.2020\/673"},{"key":"867_CR37","unstructured":"Danilevsky M, Qian K, Aharonov R, et\u00a0al (2020) A survey of the state of explainable ai for natural language processing. arXiv:2010.00711 [cs]"},{"key":"867_CR38","unstructured":"Das A, Rad P (2020) Opportunities and challenges in explainable artificial intelligence (XAI): a survey. arXiv:2006.11371"},{"issue":"1","key":"867_CR39","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1007\/s007790170019","volume":"5","author":"AK Dey","year":"2001","unstructured":"Dey AK (2001) Understanding and using context. Pers Ubiquitous Comput 5(1):4\u20137. https:\/\/doi.org\/10.1007\/s007790170019","journal-title":"Pers Ubiquitous Comput"},{"key":"867_CR40","unstructured":"Dhurandhar A, Chen PY, Luss R, et\u00a0al (2018) Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In: Advances in neural information processing systems 31. Curran Associates, Inc., pp 592\u2013603, https:\/\/proceedings.neurips.cc\/paper\/2018\/file\/c5ff2543b53f4cc0ad3819a36752467b-Paper.pdf"},{"key":"867_CR41","doi-asserted-by":"publisher","unstructured":"Donadello I, Serafini L, d\u2019Avila Garcez AS (2017) Logic tensor networks for semantic image interpretation. In: Proceedings of the 26th international joint conference on artificial intelligence. ijcai.org, pp 1596\u20131602. https:\/\/doi.org\/10.24963\/ijcai.2017\/221","DOI":"10.24963\/ijcai.2017\/221"},{"key":"867_CR42","unstructured":"Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. arXiv e-prints abs\/1702.08608"},{"key":"867_CR43","doi-asserted-by":"publisher","unstructured":"Do\u0161ilovi\u0107 FK, Br\u010di\u0107 M, Hlupi\u0107 N (2018) Explainable artificial intelligence: A survey. In: 2018 41st international convention on information and communication technology, electronics and microelectronics (MIPRO), pp 0210\u20130215. https:\/\/doi.org\/10.23919\/MIPRO.2018.8400040","DOI":"10.23919\/MIPRO.2018.8400040"},{"issue":"1","key":"867_CR44","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1145\/3359786","volume":"63","author":"M Du","year":"2019","unstructured":"Du M, Liu N, Hu X (2019) Techniques for interpretable machine learning. Commun ACM 63(1):68\u201377. https:\/\/doi.org\/10.1145\/3359786","journal-title":"Commun ACM"},{"key":"867_CR45","unstructured":"El-Assady M, Jentner W, Kehlbeck R, et\u00a0al (2019) Towards xai: structuring the processes of explanations. In: ACM workshop on human-centered machine learning"},{"key":"867_CR46","doi-asserted-by":"publisher","unstructured":"Esser P, Rombach R, Ommer B (2020) A disentangling invertible interpretation network for explaining latent representations. In: Proceedings 2020 IEEE conference on computer vision and pattern recognition. IEEE, pp 9220\u20139229. https:\/\/doi.org\/10.1109\/CVPR42600.2020.00924, https:\/\/openaccess.thecvf.com\/content_CVPR_2020\/papers\/Esser_A_Disentangling_Invertible_Interpretation_Network_for_Explaining_Latent_Representations_CVPR_2020_paper.pdf","DOI":"10.1109\/CVPR42600.2020.00924"},{"key":"867_CR47","doi-asserted-by":"crossref","unstructured":"Fails JA, Olsen\u00a0Jr DR (2003) Interactive machine learning. In: Proceedings of the 8th international conference on Intelligent user interfaces, pp 39\u201345","DOI":"10.1145\/604045.604056"},{"key":"867_CR48","doi-asserted-by":"publisher","unstructured":"Ferreira JJ, Monteiro MS (2020) What are people doing about XAI user experience? A survey on AI explainability research and practice. In: Design, user experience, and usability. Design for contemporary interactive environments. Lecture notes in computer science. Springer, pp 56\u201373. https:\/\/doi.org\/10.1007\/978-3-030-49760-6_4","DOI":"10.1007\/978-3-030-49760-6_4"},{"key":"867_CR49","unstructured":"Finzel B, Kollmann R, Rieger I, et\u00a0al (2021a) Deriving temporal prototypes from saliency map clusters for the analysis of deep-learning-based facial action unit classification. In: Seidl T, Fromm M, Obermeier S (eds) Proceedings of the LWDA 2021 Workshops: FGWM, KDML, FGWI-BIA, and FGIR, Online, September 1\u20133, 2021, CEUR Workshop Proceedings, vol 2993. CEUR-WS.org, pp 86\u201397, http:\/\/ceur-ws.org\/Vol-2993\/paper-09.pdf"},{"key":"867_CR50","doi-asserted-by":"crossref","unstructured":"Finzel B, Tafler DE, Scheele S et al (2021b) Explanation as a process: user-centric construction of multi-level and multi-modal explanations. In: Edelkamp S, M\u00f6ller R, Rueckert E (eds) KI 2021: advances in artificial intelligence. Springer, Cham, pp 80\u201394","DOI":"10.1007\/978-3-030-87626-5_7"},{"key":"867_CR51","doi-asserted-by":"publisher","unstructured":"Fong R, Vedaldi A (2018) Net2Vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In: Proceedings of the 2018 IEEE conference on computer vision and pattern recognition. IEEE Computer Society, pp 8730\u20138738. https:\/\/doi.org\/10.1109\/CVPR.2018.00910","DOI":"10.1109\/CVPR.2018.00910"},{"key":"867_CR52","doi-asserted-by":"publisher","unstructured":"Fong RC, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. In: Proceedings of the 2017 IEEE international conference on computer vision. IEEE Computer Society, pp 3449\u20133457. https:\/\/doi.org\/10.1109\/ICCV.2017.371, arXiv:1704.03296","DOI":"10.1109\/ICCV.2017.371"},{"issue":"1","key":"867_CR53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2594473.2594475","volume":"15","author":"AA Freitas","year":"2014","unstructured":"Freitas AA (2014) comprehensible classification models: a position paper. ACM SIGKDD Explor Newsl 15(1):1\u201310. https:\/\/doi.org\/10.1145\/2594473.2594475","journal-title":"ACM SIGKDD Explor Newsl"},{"issue":"5","key":"867_CR54","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1214\/aos\/1013203451","volume":"29","author":"JH Friedman","year":"2001","unstructured":"Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29(5):1189\u20131232. https:\/\/doi.org\/10.1214\/aos\/1013203451","journal-title":"Ann Stat"},{"key":"867_CR55","first-page":"9273","volume":"32","author":"A Ghorbani","year":"2019","unstructured":"Ghorbani A, Wexler J, Zou JY et al (2019) Towards automatic concept-based explanations. Adv Neural Inf Process Syst 32:9273\u20139282","journal-title":"Adv Neural Inf Process Syst"},{"key":"867_CR56","doi-asserted-by":"publisher","unstructured":"Gilpin LH, Bau D, Yuan BZ, et\u00a0al (2018) Explaining explanations: an overview of interpretability of machine learning. In: Proceedings if the 5th IEEE international conference on data science and advanced analytics. IEEE, pp 80\u201389. https:\/\/doi.org\/10.1109\/DSAA.2018.00018","DOI":"10.1109\/DSAA.2018.00018"},{"issue":"2","key":"867_CR57","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1089\/big.2016.0007","volume":"4","author":"M Gleicher","year":"2016","unstructured":"Gleicher M (2016) A framework for considering comprehensibility in modeling. Big Data 4(2):75\u201388. https:\/\/doi.org\/10.1089\/big.2016.0007","journal-title":"Big Data"},{"key":"867_CR58","doi-asserted-by":"publisher","unstructured":"Goebel R, Chander A, Holzinger K, et\u00a0al (2018) Explainable AI: the new 42? In: Machine learning and knowledge extraction. Lecture notes in computer science. Springer, pp 295\u2013303.https:\/\/doi.org\/10.1007\/978-3-319-99740-7_21","DOI":"10.1007\/978-3-319-99740-7_21"},{"issue":"1","key":"867_CR59","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1080\/10618600.2014.907095","volume":"24","author":"A Goldstein","year":"2015","unstructured":"Goldstein A, Kapelner A, Bleich J et al (2015) Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. J Comput Graph Stat 24(1):44\u201365. https:\/\/doi.org\/10.1080\/10618600.2014.907095","journal-title":"J Comput Graph Stat"},{"issue":"3","key":"867_CR60","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1609\/aimag.v38i3.2741","volume":"38","author":"B Goodman","year":"2017","unstructured":"Goodman B, Flaxman S (2017) European union regulations on algorithmic decision-making and a \u201cright to explanation\u2019\u2019. AI Mag 38(3):50\u201357. https:\/\/doi.org\/10.1609\/aimag.v38i3.2741","journal-title":"AI Mag"},{"key":"867_CR61","doi-asserted-by":"publisher","unstructured":"Guidotti R (2022) Counterfactual explanations and how to find them: literature review and benchmarking. Data Min Knowl Disc. https:\/\/doi.org\/10.1007\/s10618-022-00831-6","DOI":"10.1007\/s10618-022-00831-6"},{"issue":"5","key":"867_CR62","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti R, Monreale A, Ruggieri S et al (2018) A survey of methods for explaining black box models. ACM Comput Surv 51(5):931\u20139342. https:\/\/doi.org\/10.1145\/3236009","journal-title":"ACM Comput Surv"},{"key":"867_CR63","doi-asserted-by":"publisher","unstructured":"Guidotti R, Monreale A, Pedreschi D, et\u00a0al (2021) Principles of explainable artificial intelligence. In: Explainable AI within the digital transformation and cyber physical systems: XAI methods and applications. Springer, pp 9\u201331. https:\/\/doi.org\/10.1007\/978-3-030-76409-8_2,","DOI":"10.1007\/978-3-030-76409-8_2"},{"issue":"2","key":"867_CR64","first-page":"44","volume":"40","author":"D Gunning","year":"2019","unstructured":"Gunning D, Aha D (2019) Darpa\u2019s explainable artificial intelligence (xai) program. AI Mag 40(2):44\u201358","journal-title":"AI Mag"},{"key":"867_CR65","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aay7120","author":"D Gunning","year":"2019","unstructured":"Gunning D, Stefik M, Choi J et al (2019) XAI\u2013explainable artificial intelligence. Sci Robot. https:\/\/doi.org\/10.1126\/scirobotics.aay7120","journal-title":"Sci Robot"},{"key":"867_CR66","unstructured":"Hailesilassie T (2016) Rule extraction algorithm for deep neural networks: a review. arXiv:1610.05267"},{"issue":"1","key":"867_CR67","doi-asserted-by":"publisher","first-page":"100","DOI":"10.2307\/2346830","volume":"28","author":"JA Hartigan","year":"1979","unstructured":"Hartigan JA, Wong MA (1979) Algorithm AS 136: a k-means clustering algorithm. J R Stat Soc Ser C Appl Stat 28(1):100\u2013108. https:\/\/doi.org\/10.2307\/2346830","journal-title":"J R Stat Soc Ser C Appl Stat"},{"key":"867_CR68","doi-asserted-by":"publisher","unstructured":"Hendricks LA, Akata Z, Rohrbach M, et\u00a0al (2016) Generating visual explanations. In: Computer vision\u2013ECCV 2016. Lecture notes in computer science. Springer, pp 3\u201319. https:\/\/doi.org\/10.1007\/978-3-319-46493-0_1","DOI":"10.1007\/978-3-319-46493-0_1"},{"key":"867_CR69","doi-asserted-by":"crossref","unstructured":"Hendricks LA, Hu R, Darrell T, et\u00a0al (2018) Grounding visual explanations. In: Proceedings of the European conference on computer vision (ECCV), pp 264\u2013279","DOI":"10.1007\/978-3-030-01216-8_17"},{"key":"867_CR70","unstructured":"Henne M, Schwaiger A, Roscher K, et\u00a0al (2020) Benchmarking uncertainty estimation methods for deep learning with safety-related metrics. In: Proceedings of the workshop artificial intelligence safety, CEUR workshop proceedings, vol 2560. CEUR-WS.org, pp 83\u201390, http:\/\/ceur-ws.org\/Vol-2560\/paper35.pdf"},{"issue":"106","key":"867_CR71","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1016\/j.knosys.2020.106685","volume":"214","author":"A Heuillet","year":"2021","unstructured":"Heuillet A, Couthouis F, D\u00edaz-Rodr\u00edguez N (2021) Explainability in deep reinforcement learning. Knowl-Based Syst 214(106):685. https:\/\/doi.org\/10.1016\/j.knosys.2020.106685","journal-title":"Knowl-Based Syst"},{"issue":"1","key":"867_CR72","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/j.dss.2010.12.003","volume":"51","author":"J Huysmans","year":"2011","unstructured":"Huysmans J, Dejaeger K, Mues C et al (2011) An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decis Support Syst 51(1):141\u2013154. https:\/\/doi.org\/10.1016\/j.dss.2010.12.003","journal-title":"Decis Support Syst"},{"key":"867_CR73","doi-asserted-by":"crossref","unstructured":"Islam SR, Eberle W, Ghafoor SK, et\u00a0al (2021) Explainable artificial intelligence approaches: a survey. arXiv:2101.09429","DOI":"10.1007\/978-3-030-96630-0_4"},{"key":"867_CR74","unstructured":"ISO\/TC 22 Road vehicles (2020) ISO\/TR 4804:2020: road vehicles\u2014safety and cybersecurity for automated driving systems\u2014design, verification and validation, 1st edn. International Organization for Standardization, https:\/\/www.iso.org\/standard\/80363.html"},{"key":"867_CR75","unstructured":"ISO\/TC 22\/SC 32 (2018) ISO 26262-6:2018(En): road vehicles\u2014functional safety\u2014Part 6: product development at the software level, ISO 26262:2018(En), vol\u00a06, 2nd edn. International Organization for Standardization, https:\/\/www.iso.org\/standard\/68388.html"},{"key":"867_CR76","volume-title":"Introduction to expert systems","author":"P Jackson","year":"1998","unstructured":"Jackson P (1998) Introduction to expert systems, 3rd edn. Addison-Wesley Longman Publishing Co. Inc, New York","edition":"3"},{"key":"867_CR77","unstructured":"Janizek JD, Sturmfels P, Lee SI (2020) Explaining explanations: axiomatic feature interactions for deep networks. CoRR arXiv:2002.04138"},{"key":"867_CR78","doi-asserted-by":"publisher","unstructured":"Jolliffe IT (2002) Principal component analysis, 2nd edn. Springer Series in Statistics. Springer, https:\/\/doi.org\/10.1007\/b98835","DOI":"10.1007\/b98835"},{"key":"867_CR79","unstructured":"Karimi AH, Barthe G, Sch\u00f6lkopf B, et\u00a0al (2021) A survey of algorithmic recourse: definitions, formulations, solutions, and prospects. arXiv:2010.04050 [cs, stat]"},{"key":"867_CR80","unstructured":"Kauffmann J, Esders M, Montavon G, et\u00a0al (2019) From clustering to cluster explanations via neural networks. arXiv:1906.07633 [cs, stat]"},{"key":"867_CR81","doi-asserted-by":"publisher","unstructured":"Keane MT, Kenny EM, Delaney E, et\u00a0al (2021) If only we had better counterfactual explanations: five key deficits to rectify in the evaluation of counterfactual xai techniques. In: Twenty-ninth international joint conference on artificial intelligence, pp 4466\u20134474. https:\/\/doi.org\/10.24963\/ijcai.2021\/609","DOI":"10.24963\/ijcai.2021\/609"},{"key":"867_CR82","first-page":"5580","volume":"30","author":"A Kendall","year":"2017","unstructured":"Kendall A, Gal Y (2017) What uncertainties do we need in Bayesian deep learning for computer vision? Adv Neural Inf Process Syst 30:5580\u20135590","journal-title":"Adv Neural Inf Process Syst"},{"key":"867_CR83","doi-asserted-by":"publisher","unstructured":"Kim J, Canny JF (2017) Interpretable learning for self-driving cars by visualizing causal attention. In: Proceedings of the 2017 IEEE international conference on computer vision. IEEE Computer Society, pp 2961\u20132969. https:\/\/doi.org\/10.1109\/ICCV.2017.320","DOI":"10.1109\/ICCV.2017.320"},{"key":"867_CR84","unstructured":"Kim B, Wattenberg M, Gilmer J, et\u00a0al (2018a) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In: Proceedings of the 35th international conference on machine learning, proceedings of machine learning research, vol\u00a080. PMLR, pp 2668\u20132677, http:\/\/proceedings.mlr.press\/v80\/kim18d.html"},{"key":"867_CR85","doi-asserted-by":"publisher","unstructured":"Kim J, Rohrbach A, Darrell T, et\u00a0al (2018b) Textual explanations for self-driving vehicles. In: Proc. 15th European conference on computer vision, Part II, Lecture notes in computer science, vol 11206. Springer, pp 577\u2013593. https:\/\/doi.org\/10.1007\/978-3-030-01216-8_35, arXiv:1807.11546","DOI":"10.1007\/978-3-030-01216-8_35"},{"key":"867_CR86","unstructured":"Kindermans PJ, Sch\u00fctt KT, Alber M, et\u00a0al (2018) Learning how to explain neural networks: PatternNet and PatternAttribution. In: Proceedings of the 6th international conference on learning representations, https:\/\/openreview.net\/forum?id=Hkn7CBaTW"},{"issue":"181","key":"867_CR87","first-page":"1","volume":"22","author":"J Klaise","year":"2021","unstructured":"Klaise J, Looveren AV, Vacanti G et al (2021) Alibi explain: algorithms for explaining machine learning models. J Mach Learn Res 22(181):1\u20137","journal-title":"J Mach Learn Res"},{"key":"867_CR88","unstructured":"Koh PW, Liang P (2017) Understanding Black-box Predictions via Influence Functions. In: Proceedings of the 34th international conference on machine learning. PMLR, pp 1885\u20131894, http:\/\/proceedings.mlr.press\/v70\/koh17a.html"},{"key":"867_CR89","doi-asserted-by":"crossref","unstructured":"Kulesza T, Stumpf S, Burnett M, et\u00a0al (2010) Explanatory debugging: supporting end-user debugging of machine-learned programs. In: 2010 IEEE symposium on visual languages and human-centric computing. IEEE, pp 41\u201348","DOI":"10.1109\/VLHCC.2010.15"},{"key":"867_CR90","doi-asserted-by":"crossref","unstructured":"Kulesza T, Stumpf S, Burnett M, et\u00a0al (2013) Too much, too little, or just right? Ways explanations impact end users\u2019 mental models. In: 2013 IEEE symposium on visual languages and human centric computing. IEEE, pp 3\u201310","DOI":"10.1109\/VLHCC.2013.6645235"},{"key":"867_CR91","doi-asserted-by":"crossref","unstructured":"Kulesza T, Burnett M, Wong WK, et\u00a0al (2015) Principles of explanatory debugging to personalize interactive machine learning. In: Proceedings of the 20th international conference intelligent user interfaces, pp 126\u2013137","DOI":"10.1145\/2678025.2701399"},{"key":"867_CR92","doi-asserted-by":"publisher","unstructured":"Langer M, Oster D, Speith T, et\u00a0al (2021) What do we want from explainable artificial intelligence (XAI)?\u2014A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artif Intell, p 103473. https:\/\/doi.org\/10.1016\/j.artint.2021.103473","DOI":"10.1016\/j.artint.2021.103473"},{"issue":"1","key":"867_CR93","doi-asserted-by":"publisher","first-page":"1096","DOI":"10.1038\/s41467-019-08987-4","volume":"10","author":"S Lapuschkin","year":"2019","unstructured":"Lapuschkin S, W\u00e4ldchen S, Binder A et al (2019) Unmasking Clever Hans predictors and assessing what machines really learn. Nat Commun 10(1):1096. https:\/\/doi.org\/10.1038\/s41467-019-08987-4","journal-title":"Nat Commun"},{"key":"867_CR94","doi-asserted-by":"crossref","unstructured":"Li XH, Shi Y, Li H, et\u00a0al (2020) Quantitative evaluations on saliency methods: an experimental study. arXiv:2012.15616","DOI":"10.1145\/3447548.3467148"},{"issue":"1","key":"867_CR95","doi-asserted-by":"publisher","first-page":"18","DOI":"10.3390\/e23010018","volume":"23","author":"P Linardatos","year":"2021","unstructured":"Linardatos P, Papastefanopoulos V, Kotsiantis S (2021) Explainable AI: a review of machine learning interpretability methods. Entropy 23(1):18. https:\/\/doi.org\/10.3390\/e23010018","journal-title":"Entropy"},{"issue":"3","key":"867_CR96","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton ZC (2018) The mythos of model interpretability. Queue 16(3):31\u201357. https:\/\/doi.org\/10.1145\/3236386.3241340","journal-title":"Queue"},{"key":"867_CR97","unstructured":"Losch M, Fritz M, Schiele B (2019) Interpretability beyond classification output: semantic bottleneck networks. In: Proceedings of the 3rd ACM computer science in cars symposium extended abstracts, https:\/\/arxiv.org\/pdf\/1907.10882.pdf"},{"key":"867_CR98","first-page":"4765","volume":"30","author":"SM Lundberg","year":"2017","unstructured":"Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. Adv Neural Inf Process Syst 30:4765\u20134774","journal-title":"Adv Neural Inf Process Syst"},{"key":"867_CR99","unstructured":"Mazzine R, Martens D (2021) A framework and benchmarking study for counterfactual generating methods on tabular data. arXiv:2107.04680 [cs]"},{"key":"867_CR100","doi-asserted-by":"publisher","unstructured":"McAllister R, Gal Y, Kendall A, et\u00a0al (2017) Concrete problems for autonomous vehicle safety: advantages of Bayesian deep learning. In: Proceedings of the 26th international joint conference artificial intelligence, pp 4745\u20134753, https:\/\/doi.org\/10.24963\/ijcai.2017\/661","DOI":"10.24963\/ijcai.2017\/661"},{"key":"867_CR101","unstructured":"McCarthy J (1958) Programs with common sense. In: Proceedings of the Teddington conference on the mechanisation of thought processes, pp 77\u201384"},{"key":"867_CR102","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338. https:\/\/doi.org\/10.1016\/j.artint.2018.07.007","journal-title":"Artif Intell"},{"key":"867_CR103","unstructured":"Molnar C (2020) Interpretable machine learning. Lulu.com, https:\/\/christophm.github.io\/interpretable-ml-book\/"},{"key":"867_CR104","unstructured":"Muddamsetty SM, Jahromi MNS, Ciontos AE, et\u00a0al (2021) Introducing and assessing the explainable AI (XAI) method: SIDU. arXiv:2101.10710"},{"key":"867_CR105","unstructured":"Mueller ST, Hoffman RR, Clancey W, et\u00a0al (2019) Explanation in human-AI systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable AI. arXiv:1902.01876"},{"key":"867_CR106","unstructured":"Mueller ST, Veinott ES, Hoffman RR, et\u00a0al (2021) Principles of explanation in human-AI systems. CoRR arXiv:2102.04972"},{"issue":"7","key":"867_CR107","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1007\/s10994-018-5707-3","volume":"107","author":"SH Muggleton","year":"2018","unstructured":"Muggleton SH, Schmid U, Zeller C et al (2018) Ultra-strong machine learning: comprehensibility of programs learned with ilp. Mach Learn 107(7):1119\u20131140","journal-title":"Mach Learn"},{"issue":"44","key":"867_CR108","doi-asserted-by":"publisher","first-page":"22071","DOI":"10.1073\/pnas.1900654116","volume":"116","author":"WJ Murdoch","year":"2019","unstructured":"Murdoch WJ, Singh C, Kumbier K et al (2019) Definitions, methods, and applications in interpretable machine learning. PNAS 116(44):22071\u201322080. https:\/\/doi.org\/10.1073\/pnas.1900654116","journal-title":"PNAS"},{"key":"867_CR109","doi-asserted-by":"publisher","unstructured":"Nguyen A, Yosinski J, Clune J (2019) Understanding neural networks via feature visualization: a survey. In: Explainable AI: interpreting, explaining and visualizing deep learning. Lecture notes in computer science. Springer, pp 55\u201376. https:\/\/doi.org\/10.1007\/978-3-030-28954-6_4","DOI":"10.1007\/978-3-030-28954-6_4"},{"key":"867_CR110","unstructured":"Nori H, Jenkins S, Koch P, et\u00a0al (2019) InterpretML: a unified framework for machine learning interpretability. CoRR arXiv:1909.09223"},{"issue":"3\u20135","key":"867_CR111","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1007\/s11257-017-9195-0","volume":"27","author":"I Nunes","year":"2017","unstructured":"Nunes I, Jannach D (2017) A systematic review and taxonomy of explanations in decision support and recommender systems. User Model User Adap Inter 27(3\u20135):393\u2013444. https:\/\/doi.org\/10.1007\/s11257-017-9195-0","journal-title":"User Model User Adap Inter"},{"issue":"11","key":"867_CR112","doi-asserted-by":"publisher","DOI":"10.23915\/distill.00007","volume":"2","author":"C Olah","year":"2017","unstructured":"Olah C, Mordvintsev A, Schubert L (2017) Feature visualization. Distill 2(11):e7. https:\/\/doi.org\/10.23915\/distill.00007","journal-title":"Feature visualization. Distill"},{"issue":"3","key":"867_CR113","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1007\/s11023-019-09502-w","volume":"29","author":"A P\u00e1ez","year":"2019","unstructured":"P\u00e1ez A (2019) The pragmatic turn in explainable artificial intelligence (xai). Mind Mach 29(3):441\u2013459","journal-title":"Mind Mach"},{"key":"867_CR114","unstructured":"Petsiuk V, Das A, Saenko K (2018) RISE: randomized input sampling for explanation of black-box models. In: Proceedings of the British machine vision conference. BMVA Press, p 151, http:\/\/bmvc2018.org\/contents\/papers\/1064.pdf"},{"key":"867_CR115","doi-asserted-by":"crossref","unstructured":"Petsiuk V, Jain R, Manjunatha V, et\u00a0al (2021) Black-box explanation of object detectors via saliency maps. In: Proceedings of the 2021 IEEE\/CVF conference on computer vision and pattern recognition, pp 11443\u201311452, https:\/\/openaccess.thecvf.com\/content\/CVPR2021\/html\/Petsiuk_Black-Box_Explanation_of_Object_Detectors_via_Saliency_Maps_CVPR_2021_paper.html","DOI":"10.1109\/CVPR46437.2021.01128"},{"key":"867_CR116","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1007\/978-3-030-50402-1_4","volume":"2020","author":"M Pocevi\u010di\u016bt\u0117","year":"2020","unstructured":"Pocevi\u010di\u016bt\u0117 M, Eilertsen G, Lundstr\u00f6m C (2020) Survey of XAI in digital pathology. Lect Notes Comput Sci 2020:56\u201388. https:\/\/doi.org\/10.1007\/978-3-030-50402-1_4","journal-title":"Lect Notes Comput Sci"},{"key":"867_CR117","doi-asserted-by":"publisher","unstructured":"Puiutta E, Veith EMSP (2020) Explainable reinforcement learning: a survey. In: Machine learning and knowledge extraction\u20144th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 international cross-domain conference, CD-MAKE 2020, Dublin, Ireland, August 25\u201328, 2020, Proceedings, lecture notes in computer science, vol 12279. Springer, pp 77\u201395, https:\/\/doi.org\/10.1007\/978-3-030-57321-8_5","DOI":"10.1007\/978-3-030-57321-8_5"},{"key":"867_CR118","unstructured":"Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann series in machine learning. Morgan Kaufmann, https:\/\/kupdf.net\/download\/j-ross-quinlan-c4-5-programs-for-machine-learning-1993_5b095daee2b6f5024deefc30_pdf"},{"key":"867_CR119","doi-asserted-by":"publisher","unstructured":"Rabold J, Siebers M, Schmid U (2018) Explaining black-box classifiers with ILP\u2014empowering LIME with Aleph to approximate non-linear decisions with relational rules. In: International conference on machine inductive logic programming. Lecture notes in computer science. Springer, pp 105\u2013117. https:\/\/doi.org\/10.1007\/978-3-319-99960-9_7,","DOI":"10.1007\/978-3-319-99960-9_7"},{"key":"867_CR120","doi-asserted-by":"publisher","unstructured":"Rabold J, Schwalbe G, Schmid U (2020) Expressive explanations of DNNs by combining concept analysis with ILP. In: KI 2020: advances in artificial intelligence. Lecture notes in computer science. Springer, pp 148\u2013162. https:\/\/doi.org\/10.1007\/978-3-030-58285-2_11","DOI":"10.1007\/978-3-030-58285-2_11"},{"key":"867_CR121","unstructured":"Renard X, Woloszko N, Aigrain J, et\u00a0al (2019) Concept tree: high-level representation of variables for more interpretable surrogate decision trees. In: Proceedings of the 2019 ICML workshop human in the loop learning, arXiv:1906.01297"},{"key":"867_CR122","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \"Why should I trust you?\": explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International conference on knowledge discovery and data mining. ACM, KDD\u201916, pp 1135\u20131144, arXiv:1602.04938","DOI":"10.1145\/2939672.2939778"},{"key":"867_CR123","unstructured":"Rieger I, Kollmann R, Finzel B, et\u00a0al (2020) Verifying deep learning-based decisions for facial expression recognition. In: Proceedings of the ESANN conference 2020"},{"issue":"5","key":"867_CR124","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","volume":"1","author":"C Rudin","year":"2019","unstructured":"Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1(5):206\u2013215. https:\/\/doi.org\/10.1038\/s42256-019-0048-x","journal-title":"Nat Mach Intell"},{"key":"867_CR125","first-page":"3856","volume":"30","author":"S Sabour","year":"2017","unstructured":"Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. Adv Neural Inf Process Syst 30:3856\u20133866","journal-title":"Adv Neural Inf Process Syst"},{"key":"867_CR126","doi-asserted-by":"publisher","unstructured":"Saeed W, Omlin C (2021) Explainable AI (XAI): a systematic meta-survey of current challenges and future opportunities. https:\/\/doi.org\/10.48550\/arXiv.2111.06420","DOI":"10.48550\/arXiv.2111.06420"},{"key":"867_CR127","doi-asserted-by":"publisher","unstructured":"Samek W, M\u00fcller KR (2019) Towards explainable artificial intelligence. In: Explainable AI: interpreting, explaining and visualizing deep learning, Lecture notes in computer science, vol 11700. Springer, p 5\u201322. https:\/\/doi.org\/10.1007\/978-3-030-28954-6_1","DOI":"10.1007\/978-3-030-28954-6_1"},{"key":"867_CR128","doi-asserted-by":"publisher","unstructured":"Samek W, Montavon G, Vedaldi A et al (2019) Explainable AI: interpreting, explaining and visualizing deep learning, Lecture notes in computer science, vol 11700. Springer. https:\/\/doi.org\/10.1007\/978-3-030-28954-6","DOI":"10.1007\/978-3-030-28954-6"},{"key":"867_CR129","unstructured":"Samek W, Montavon G, Lapuschkin S, et\u00a0al (2020) Toward interpretable machine learning: transparent deep neural networks and beyond. CoRR arXiv:2003.07631"},{"key":"867_CR130","doi-asserted-by":"crossref","unstructured":"Schmid U, Finzel B (2020) Mutual explanations for cooperative decision making in medicine. KI-K\u00fcnstliche Intelligenz pp 227\u2013233","DOI":"10.1007\/s13218-020-00633-2"},{"key":"867_CR131","doi-asserted-by":"crossref","unstructured":"Schmid U, Zeller C, Besold T, et\u00a0al (2016) How does predicate invention affect human comprehensibility? In: International conference on inductive logic programming. Springer, pp 52\u201367","DOI":"10.1007\/978-3-319-63342-8_5"},{"key":"867_CR132","doi-asserted-by":"publisher","unstructured":"Selvaraju RR, Cogswell M, Das A, et\u00a0al (2017) Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the 2017 IEEE international conference on computer vision. IEEE, pp 618\u2013626, https:\/\/doi.org\/10.1109\/ICCV.2017.74, https:\/\/arxiv.org\/abs\/1610.02391","DOI":"10.1109\/ICCV.2017.74"},{"key":"867_CR133","unstructured":"Shwartz-Ziv R, Tishby N (2017) Opening the black box of deep neural networks via information. CoRR arXiv:1703.00810"},{"key":"867_CR134","unstructured":"Simonyan K, Vedaldi A, Zisserman A (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. In: Proceedings of the 2nd international conference on learning representations, workshop track proceedings, arXiv:1312.6034"},{"issue":"6","key":"867_CR135","doi-asserted-by":"publisher","first-page":"52","DOI":"10.3390\/jimaging6060052","volume":"6","author":"A Singh","year":"2020","unstructured":"Singh A, Sengupta S, Lakshminarayanan V (2020) Explainable deep learning models in medical image analysis. J Imaging 6(6):52. https:\/\/doi.org\/10.3390\/jimaging6060052","journal-title":"J Imaging"},{"key":"867_CR136","unstructured":"Smilkov D, Thorat N, Kim B, et\u00a0al (2017) SmoothGrad: removing noise by adding noise. CoRR arXiv:1706.03825"},{"issue":"49","key":"867_CR137","doi-asserted-by":"publisher","first-page":"1904","DOI":"10.21105\/joss.01904","volume":"5","author":"K Sokol","year":"2020","unstructured":"Sokol K, Hepburn A, Poyiadzi R et al (2020) Fat forensics: a python toolbox for implementing and deploying fairness, accountability and transparency algorithms in predictive systems. J Open Source Softw 5(49):1904. https:\/\/doi.org\/10.21105\/joss.01904","journal-title":"J Open Source Softw"},{"key":"867_CR138","doi-asserted-by":"publisher","first-page":"1064","DOI":"10.1109\/TVCG.2019.2934629","volume":"26","author":"T Spinner","year":"2020","unstructured":"Spinner T, Schlegel U, Schafer H et al (2020) explAIner: a visual analytics framework for interactive and explainable machine learning. IEEE Trans Vis Comput Gr 26:1064\u20131074. https:\/\/doi.org\/10.1109\/TVCG.2019.2934629","journal-title":"IEEE Trans Vis Comput Gr"},{"key":"867_CR139","unstructured":"Springenberg JT, Dosovitskiy A, Brox T, et\u00a0al (2015) Striving for simplicity: the all convolutional net. In: Proceedings of the 3rd international conference on learning representations, ICLR 2015, workshop track proceedings, arXiv:1412.6806"},{"key":"867_CR140","doi-asserted-by":"publisher","first-page":"11974","DOI":"10.1109\/ACCESS.2021.3051315","volume":"9","author":"I Stepin","year":"2021","unstructured":"Stepin I, Alonso JM, Catala A et al (2021) A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence. IEEE Access 9:11974\u201312001. https:\/\/doi.org\/10.1109\/ACCESS.2021.3051315","journal-title":"IEEE Access"},{"key":"867_CR141","unstructured":"Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: Proceedings of the 34th international conference on machin learning, proceedings of machine learning research, vol\u00a070. PMLR, pp 3319\u20133328, http:\/\/proceedings.mlr.press\/v70\/sundararajan17a.html"},{"key":"867_CR142","doi-asserted-by":"crossref","unstructured":"Teso S, Kersting K (2019) Explanatory interactive machine learning. In: Proceedings of the 2019 AAAI\/ACM conference on AI, ethics, and society, pp 239\u2013245","DOI":"10.1145\/3306618.3314293"},{"key":"867_CR143","first-page":"505","volume":"7","author":"S Thrun","year":"1995","unstructured":"Thrun S (1995) Extracting rules from artificial neural networks with distributed representations. Adv Neural Inf Process Syst 7:505\u2013512","journal-title":"Adv Neural Inf Process Syst"},{"key":"867_CR144","doi-asserted-by":"publisher","unstructured":"Tintarev N, Masthoff J (2007) A survey of explanations in recommender systems. In: IEEE 23rd international conference on data engineering workshop, pp 801\u2013810. https:\/\/doi.org\/10.1109\/ICDEW.2007.4401070","DOI":"10.1109\/ICDEW.2007.4401070"},{"key":"867_CR145","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.3027314","author":"E Tjoa","year":"2020","unstructured":"Tjoa E, Guan C (2020) A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans Neural Netw Learn Syst. https:\/\/doi.org\/10.1109\/TNNLS.2020.3027314","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"issue":"86","key":"867_CR146","first-page":"2579","volume":"9","author":"L van der Maaten","year":"2008","unstructured":"van der Maaten L, Hinton G (2008) Visualizing Data using t-SNE. J Mach Learn Res 9(86):2579\u20132605","journal-title":"J Mach Learn Res"},{"key":"867_CR147","unstructured":"van Lent M, Fisher W, Mancuso M (2004) An explainable artificial intelligence system for small-unit tactical behavior. In: Proceedings of the 2004 national conference artificial intelligence. AAAI Press; 1999, pp 900\u2013907"},{"key":"867_CR148","doi-asserted-by":"publisher","unstructured":"Vassiliades A, Bassiliades N, Patkos T (2021\/ed) Argumentation and explainable artificial intelligence: a survey. Knowl Eng Rev. https:\/\/doi.org\/10.1017\/S0269888921000011","DOI":"10.1017\/S0269888921000011"},{"key":"867_CR149","unstructured":"Verma S, Dickerson J, Hines K (2020) Counterfactual explanations for machine learning: a review. arXiv:2010.10596 [cs, stat]"},{"key":"867_CR150","unstructured":"Vilone G, Longo L (2020) Explainable artificial intelligence: a systematic review. arXiv:2006.00093"},{"issue":"4","key":"867_CR151","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1007\/s11222-007-9033-z","volume":"17","author":"U von Luxburg","year":"2007","unstructured":"von Luxburg U (2007) A tutorial on spectral clustering. Stat Comput 17(4):395\u2013416. https:\/\/doi.org\/10.1007\/s11222-007-9033-z","journal-title":"Stat Comput"},{"key":"867_CR152","unstructured":"Wan A, Dunlap L, Ho D, et\u00a0al (2020) NBDT: neural-backed decision tree. In: Posters 2021 international conference on learning representations, https:\/\/openreview.net\/forum?id=mCLVeEpplNE"},{"key":"867_CR153","doi-asserted-by":"publisher","unstructured":"Wang H (2018) ReNN: rule-embedded neural networks. In: Proceedings of the 24th international conference on pattern recognition. IEEE Computer Society, pp 824\u2013829. https:\/\/doi.org\/10.1109\/ICPR.2018.8545379, http:\/\/arxiv.org\/abs\/1801.09856","DOI":"10.1109\/ICPR.2018.8545379"},{"key":"867_CR154","doi-asserted-by":"crossref","unstructured":"Wang Q, Zhang K, II AGO, et\u00a0al (2018a) A comparative study of rule extraction for recurrent neural networks. CoRR arXiv:1801.05420","DOI":"10.1162\/neco_a_01111"},{"key":"867_CR155","doi-asserted-by":"publisher","unstructured":"Wang Q, Zhang K, Ororbia AG II et al (2018b) An empirical evaluation of rule extraction from recurrent neural networks. Neural Comput 30(9):2568\u20132591. https:\/\/doi.org\/10.1162\/neco_a_01111","DOI":"10.1162\/neco_a_01111"},{"key":"867_CR156","unstructured":"Weitz K (2018) Applying explainable artificial intelligence for deep learning networks to decode facial expressions of pain and emotions. Master\u2019s thesis, Otto-Friedrich-University Bamberg, http:\/\/www.cogsys.wiai.uni-bamberg.de\/theses\/weitz\/Masterarbeit_Weitz.pdf"},{"key":"867_CR157","unstructured":"Xie N, Ras G, van Gerven M, et\u00a0al (2020) Explainable deep learning: a field guide for the uninitiated. CoRR arXiv:2004.14545"},{"key":"867_CR158","unstructured":"Yang Y, Feng C, Shen Y, et\u00a0al (2017) Foldingnet: interpretable unsupervised learning on 3d point clouds. CoRR arXiv:1712.07262"},{"key":"867_CR159","doi-asserted-by":"publisher","unstructured":"Yao J (2005) Knowledge extracted from trained neural networks: What\u2019s next? In: Data mining, intrusion detection, information assurance, and data networks security 2005, Orlando, Florida, USA, March 28\u201329, 2005. SPIE Proceedings, vol 5812. SPIE, pp 151\u2013157. https:\/\/doi.org\/10.1117\/12.604463","DOI":"10.1117\/12.604463"},{"key":"867_CR160","first-page":"20554","volume":"33","author":"CK Yeh","year":"2020","unstructured":"Yeh CK, Kim B, Arik S et al (2020) On completeness-aware concept-based explanations in deep neural networks. Adv Neural Inf Process Syst 33:20554\u201320565","journal-title":"Adv Neural Inf Process Syst"},{"key":"867_CR161","doi-asserted-by":"publisher","unstructured":"Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Proceedings of the 13th European conference on computer vision\u2014part I, lecture notes in computer science, vol 8689. Springer, pp 818\u2013833, https:\/\/doi.org\/10.1007\/978-3-319-10590-1_53,","DOI":"10.1007\/978-3-319-10590-1_53"},{"issue":"1","key":"867_CR162","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1631\/FITEE.1700808","volume":"19","author":"Q Zhang","year":"2018","unstructured":"Zhang Q, Zhu SC (2018) Visual interpretability for deep learning: a survey. Front IT EE 19(1):27\u201339. https:\/\/doi.org\/10.1631\/FITEE.1700808","journal-title":"Front IT EE"},{"issue":"1","key":"867_CR163","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/1500000066","volume":"14","author":"Y Zhang","year":"2020","unstructured":"Zhang Y, Chen X (2020) Explainable recommendation: a survey and new perspectives. FNT Inf Retr 14(1):1\u2013101. https:\/\/doi.org\/10.1561\/1500000066","journal-title":"FNT Inf Retr"},{"key":"867_CR164","doi-asserted-by":"crossref","unstructured":"Zhang Q, Cao R, Shi F, et\u00a0al (2018) Interpreting CNN knowledge via an explanatory graph. In: Proceedings of the 32nd AAAI conference on artificial intelligence. AAAI Press, pp 4454\u20134463, https:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI18\/paper\/view\/17354","DOI":"10.1609\/aaai.v32i1.11819"},{"key":"867_CR165","doi-asserted-by":"publisher","unstructured":"Zhang Y, Ti\u0148o P, Leonardis A, Tang K (2021) A survey on neural network interpretability. IEEE Trans Emerg Top Comput Intell 5(5):726\u2013742. https:\/\/doi.org\/10.1109\/TETCI.2021.3100641","DOI":"10.1109\/TETCI.2021.3100641"},{"key":"867_CR166","doi-asserted-by":"publisher","unstructured":"Zhou B, Khosla A, Lapedriza \u00c0, et\u00a0al (2016) Learning deep features for discriminative localization. In: Proceedings of the 2016 IEEE conference computer vision and pattern recognition. IEEE Computer Society, pp 2921\u20132929. https:\/\/doi.org\/10.1109\/CVPR.2016.319, arXiv:1512.04150","DOI":"10.1109\/CVPR.2016.319"},{"key":"867_CR167","doi-asserted-by":"publisher","unstructured":"Zhou B, Sun Y, Bau D, et\u00a0al (2018) Interpretable basis decomposition for visual explanation. In: Computer vision\u2014ECCV 2018. Lecture notes in computer science. Springer, pp 122\u2013138. https:\/\/doi.org\/10.1007\/978-3-030-01237-3_8","DOI":"10.1007\/978-3-030-01237-3_8"},{"issue":"5","key":"867_CR168","doi-asserted-by":"publisher","first-page":"593","DOI":"10.3390\/electronics10050593","volume":"10","author":"J Zhou","year":"2021","unstructured":"Zhou J, Gandomi AH, Chen F et al (2021) Evaluating the quality of machine learning explanations: a survey on methods and metrics. Electronics 10(5):593. https:\/\/doi.org\/10.3390\/electronics10050593","journal-title":"Electronics"},{"key":"867_CR169","doi-asserted-by":"publisher","unstructured":"Zilke JR, Loza\u00a0Menc\u00eda E, Janssen F (2016) DeepRED\u2014rule extraction from deep neural networks. In: Proceedings of the 19th international conference discovery science, Lecture notes in computer science. Springer, pp 457\u2013473. https:\/\/doi.org\/10.1007\/978-3-319-46307-0_29,","DOI":"10.1007\/978-3-319-46307-0_29"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00867-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-022-00867-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00867-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,31]],"date-time":"2024-08-31T11:10:10Z","timestamp":1725102610000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-022-00867-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,6]]},"references-count":169,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["867"],"URL":"https:\/\/doi.org\/10.1007\/s10618-022-00867-8","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,6]]},"assertion":[{"value":"31 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 January 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This work was authored in the scope of doctoral research of both authors, both supervised by Prof.\u00a0Dr.\u00a0Ute Schmid at the University of Bamberg. The authors declare that they have no competing financial interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}}]}}