{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T07:03:31Z","timestamp":1773903811765,"version":"3.50.1"},"reference-count":96,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2022,12,1]],"date-time":"2022-12-01T00:00:00Z","timestamp":1669852800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T00:00:00Z","timestamp":1670371200000},"content-version":"vor","delay-in-days":6,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005721","name":"Universit\u00e4t Bielefeld","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005721","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With the perspective on applications of AI-technology, especially data intensive deep learning approaches, the need for methods to control and understand such models has been recognized and gave rise to a new research domain labeled explainable artificial intelligence (XAI). In this overview paper we give an interim appraisal of what has been achieved so far and where there are still gaps in the research. We take an interdisciplinary perspective to identify challenges on XAI research and point to open questions with respect to the quality of the explanations regarding faithfulness and consistency of explanations. On the other hand we see a need regarding the interaction between XAI and user to allow for adaptability to specific information needs and explanatory dialog for informed decision making as well as the possibility to correct models and explanations by interaction. This endeavor requires an integrated interdisciplinary perspective and rigorous approaches to empirical evaluation based on psychological, linguistic and even sociological theories.<\/jats:p>","DOI":"10.1007\/s13218-022-00786-2","type":"journal-article","created":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T13:04:20Z","timestamp":1670418260000},"page":"303-315","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["What is Missing in XAI So Far?"],"prefix":"10.1007","volume":"36","author":[{"given":"Ute","family":"Schmid","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1424-472X","authenticated-orcid":false,"given":"Britta","family":"Wrede","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,12,7]]},"reference":[{"key":"786_CR1","doi-asserted-by":"publisher","first-page":"102887","DOI":"10.1016\/j.ijhcs.2022.102887","volume":"167","author":"R Abdelghanisps","year":"2022","unstructured":"Abdelghanisps R, Pierre-Yves O, Edith L, Catherine V, H\u00e9l\u00e8ne S (2022) Conversational agents for fostering curiosity-driven learning in children. Int J Hum Comput Stud 167:102887","journal-title":"Int J Hum Comput Stud"},{"key":"786_CR2","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE Access 6:52138\u201352160","journal-title":"IEEE Access"},{"issue":"4","key":"786_CR3","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1007\/s10994-020-05941-0","volume":"110","author":"L Ai","year":"2021","unstructured":"Ai L, Stephen HM, C\u00e9line H, Mark G, Ute S (2021) Beneficial and harmful explanatory machine learning. Mach Learn 110(4):695\u2013721","journal-title":"Mach Learn"},{"issue":"1","key":"786_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2021.103581","volume":"25","author":"RA Arjun","year":"2022","unstructured":"Arjun RA, Keze W, Changsong L, Sari S-S, Hongjing L, Sinisa T, Joyce C, Song-Chun Z (2022) CX-ToM: counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models. Iscience 25(1):103581","journal-title":"Iscience"},{"key":"786_CR5","unstructured":"Ali A, Jugal KK (2018) Intelligent tutoring systems: a comprehensive historical survey with recent developments. arXiv preprint arXiv:1812.09628"},{"issue":"2","key":"786_CR6","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/0890-5401(87)90052-6","volume":"75","author":"D Angluin","year":"1987","unstructured":"Angluin D (1987) Learning regular sets from queries and counterexamples. Inf Comput 75(2):87\u2013106","journal-title":"Inf Comput"},{"key":"786_CR7","doi-asserted-by":"crossref","unstructured":"Atanasove P, Jakob\u00a0GS, Christina L, Isabelle A (2020) A diagnostic study of explainability techniques for text classification. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, pp 3256\u20133274","DOI":"10.18653\/v1\/2020.emnlp-main.263"},{"issue":"12","key":"786_CR8","doi-asserted-by":"publisher","first-page":"1226","DOI":"10.1038\/s41592-019-0582-9","volume":"16","author":"S Berg","year":"2019","unstructured":"Berg S, Kutra D, Kroeger T, Straehle CN, Kausler BX, Haubold C, Schiegg M, Ales J, Beier T, Rudy M et al (2019) Ilastik: interactive machine learning for (bio) image analysis. Nat Methods 16(12):1226\u20131232","journal-title":"Nat Methods"},{"issue":"1","key":"786_CR9","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1007\/s13218-015-0414-8","volume":"30","author":"S Biundo","year":"2016","unstructured":"Biundo S, Wendemuth A (2016) Companion-technology for cognitive technical systems. K\u00fcnstliche Intell 30(1):71\u201375","journal-title":"K\u00fcnstliche Intell"},{"key":"786_CR10","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2020.507973","volume":"3","author":"S Bruckert","year":"2020","unstructured":"Bruckert S, Finzel B, Schmid U (2020) The next generation of medical decision support: a roadmap toward transparent expert companions. Front Artif Intell 3:507973","journal-title":"Front Artif Intell"},{"issue":"1","key":"786_CR11","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1177\/026565908500100113","volume":"1","author":"J Bruner","year":"1985","unstructured":"Bruner J (1985) Child\u2019s talk: learning to use language. Child Lang Teach Therapy 1(1):111\u2013114","journal-title":"Child Lang Teach Therapy"},{"key":"786_CR12","doi-asserted-by":"crossref","unstructured":"Brusilovsky P, de Marco G, Alexander F, Pasquale L, Marco P, Giovanni S, Martijn\u00a0CW (2022) Joint workshop on interfaces and human decision making for recommender systems (IntRS\u201922). In: Proceedings of the 16th ACM Conference on Recommender Systems, RecSys\u201922, New York, NY, USA. Association for Computing Machinery, pp 667\u2013670","DOI":"10.1145\/3523227.3547413"},{"key":"786_CR13","unstructured":"Buschmeier H, Stefan K (2014) A dynamic minimal model of the listener for feedback-based dialogue coordination. In: Proceedings of SemDial 2014, Workshop on the Semantics and Pragmatics of Dialogue"},{"key":"786_CR14","doi-asserted-by":"crossref","unstructured":"Bussone A, Simone S, Dympna OS (2015) The role of explanations on trust and reliance in clinical decision support systems. In: 2015 International conference on healthcare informatics, IEEE, pp 160\u2013169","DOI":"10.1109\/ICHI.2015.26"},{"key":"786_CR15","doi-asserted-by":"crossref","unstructured":"Chen R, Hao C, Ge\u00a0H, Jie R, Quanshi Z (2019) Explaining neural networks semantically and quantitatively. In: 2019 IEEE\/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), IEEE, pp 9186\u20139195","DOI":"10.1109\/ICCV.2019.00928"},{"key":"786_CR16","doi-asserted-by":"publisher","first-page":"09","DOI":"10.1207\/s1532690xci2203_4","volume":"22","author":"M Chi","year":"2004","unstructured":"Chi M, Siler S, Jeong H (2004) Can tutors monitor students understanding accurately? Cogn Instr 22:09","journal-title":"Cogn Instr"},{"issue":"4","key":"786_CR17","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1080\/00461520.2014.965823","volume":"49","author":"MTH Chi","year":"2014","unstructured":"Chi MTH, Wylie R (2014) The ICAP framework: linking cognitive engagement to active learning outcomes. Educ Psychol 49(4):219\u2013243","journal-title":"Educ Psychol"},{"issue":"3","key":"786_CR18","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/0004-3702(83)90008-5","volume":"20","author":"WJ Clancey","year":"1983","unstructured":"Clancey WJ (1983) The epistemology of a rule-based expert system-a framework for explanation. Artif Intell 20(3):215\u2013251","journal-title":"Artif Intell"},{"key":"786_CR19","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1207\/s15327809jls0604_1","volume":"6","author":"EB Coleman","year":"1997","unstructured":"Coleman EB, Brown AL, Rivkin ID (1997) The effect of instructional explanations on learning from scientific texts. J Learn Sci 6:347\u2013365","journal-title":"J Learn Sci"},{"key":"786_CR20","first-page":"8","volume-title":"Advances in neural information processing systems","author":"M Craven","year":"1995","unstructured":"Craven M, Jude S (1995) Extracting tree-structured representations of trained networks. Advances in neural information processing systems. Springer, Cham, p 8"},{"key":"786_CR21","first-page":"32","volume-title":"Advances in neural information processing systems","author":"W-Z Dai","year":"2019","unstructured":"Dai W-Z, Qiuling X, Yang Y, Zhi-Hua Z (2019) Bridging machine learning and logical reasoning by abductive learning. Advances in neural information processing systems. Springer, Cham, p 32"},{"key":"786_CR22","first-page":"590","volume-title":"Advances in neural information processing systems: annual conference on neural information processing systems","author":"A Dhurandhar","year":"2018","unstructured":"Dhurandhar A, Pin-Yu C, Ronny L, Chun-Chen T, Pai-Shun T, Karthikeyan S, Payel D (2018) Explanations based on the missing: towards contrastive explanations with pertinent negatives. In: Samy B, Hanna MW, Hugo L, Kristen G, Nicol\u00f2 C-B, Roman G (eds) Advances in neural information processing systems: annual conference on neural information processing systems. NeurIPS, Montr\u00e9al, pp 590\u2013601"},{"key":"786_CR23","unstructured":"Doshi F-V, Been K (2017) Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608"},{"key":"786_CR24","unstructured":"El-Assady M, Wolfgang J, Rebecca K, Udo S, Rita S, Fabian S, Thilo S, Daniel K (2019) Towards XAI: structuring the processes of explanations. In: Proceedings of HCML Workshop at CHI\u201919"},{"key":"786_CR25","doi-asserted-by":"crossref","unstructured":"Fails\u00a0JF, Dan\u00a0RO Jr (2003) Interactive machine learning. In: Proceedings of the 8th International Conference on Intelligent User Interfaces, pp 39\u201345","DOI":"10.1145\/604045.604056"},{"key":"786_CR26","first-page":"80","volume-title":"KI 2021: advances in artificial intelligence - 44th German conference on AI, virtual event, September 27 - October 1, 2021, proceedings, volume 12873 of lecture notes in computer science","author":"B Finzel","year":"2021","unstructured":"Finzel B, David ET, Stephan S, Ute S (2021) Explanation as a process: user-centric construction of multi-level and multi-modal explanations. In: Stefan ER, Elmar R (eds) KI 2021: advances in artificial intelligence - 44th German conference on AI, virtual event, September 27 - October 1, 2021, proceedings, volume 12873 of lecture notes in computer science. Springer, Cham, pp 80\u201394"},{"key":"786_CR27","unstructured":"Finzel B, David\u00a0ET, Anna\u00a0MT, Ute S (2021) Multimodal explanations for user-centric medical decision support systems. In: HUMAN@AAAI Fall Symposium"},{"issue":"3","key":"786_CR28","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1111\/j.1467-9280.1994.tb00652.x","volume":"5","author":"D Gentner","year":"1994","unstructured":"Gentner D, Markman AB (1994) Structural alignment in comparison: no difference without similarity. Psychol Sci 5(3):152\u2013158","journal-title":"Psychol Sci"},{"key":"786_CR29","first-page":"210","volume-title":"Artificial intelligence in education","author":"M Glass","year":"2001","unstructured":"Glass M (2001) Processing language input in the CIRCSIM-tutor intelligent tutoring system. Artificial intelligence in education. IOS Press, Amsterdam, pp 210\u2013221"},{"key":"786_CR30","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1002\/(SICI)1098-2736(199901)36:1<39::AID-TEA4>3.0.CO;2-I","volume":"36","author":"JD Gobert","year":"1999","unstructured":"Gobert JD, Clement JJ (1999) Effects of student-generated dia- grams versus student-generated summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. J Res Sci Teach 36:39\u201353","journal-title":"J Res Sci Teach"},{"key":"786_CR31","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1007\/978-3-319-99740-7_21","volume-title":"International cross-domain conference for machine learning and knowledge extraction","author":"R Goebel","year":"2018","unstructured":"Goebel R, Chander A, Holzinger K, Lecue F, Akata Z, Stumpf S, Kieseberg P, Holzinger A (2018) Explainable AI: the new 42? International cross-domain conference for machine learning and knowledge extraction. Springer, Cham, pp 295\u2013303"},{"key":"786_CR32","doi-asserted-by":"crossref","unstructured":"Guerdan L, Alex R, Hatice G (2021) Toward affective XAI: facial affect analysis for understanding explainable human-AI interactions. In: Proceedings of of the IEEE\/CVF International Conference on Computer Vision, pp 3796\u20133805","DOI":"10.1109\/ICCVW54120.2021.00423"},{"issue":"2","key":"786_CR33","first-page":"44","volume":"40","author":"D Gunning","year":"2019","unstructured":"Gunning D, Aha D (2019) DARPA\u2019s explainable artificial intelligence (XAI) program. AI Mag 40(2):44\u201358","journal-title":"AI Mag"},{"issue":"37","key":"786_CR34","doi-asserted-by":"publisher","first-page":"eaay7120","DOI":"10.1126\/scirobotics.aay7120","volume":"4","author":"D Gunning","year":"2019","unstructured":"Gunning D, Stefik M, Choi J, Miller T, Stumpf S, Yang G-Z (2019) XAI - explainable artificial intelligence. Sci Robot 4(37):eaay7120","journal-title":"Sci Robot"},{"key":"786_CR35","first-page":"332","volume-title":"Proceedings of the 2017 ACM\/IEEE international conference on human-robot interaction, HRI","author":"J Hemminghaus","year":"2017","unstructured":"Hemminghaus J, Stefan K (2017) Towards adaptive social behavior generation for assistive robots using reinforcement learning. In: Bilge M, Manfred T, Astrid W, James EY (eds) Proceedings of the 2017 ACM\/IEEE international conference on human-robot interaction, HRI. ACM, Vienna, pp 332\u2013340"},{"key":"786_CR36","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1086\/286983","volume":"15","author":"CG Hempel","year":"1948","unstructured":"Hempel CG, Oppenheim P (1948) Studies in the logic of explanation. Philos Sci 15:135\u201375","journal-title":"Philos Sci"},{"key":"786_CR37","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1007\/978-3-031-09282-4_35","volume-title":"International conference on pattern recognition and artificial intelligence (ICPRAI)","author":"M Herchenbach","year":"2022","unstructured":"Herchenbach M, M\u00fcller D, Scheele S, Schmid U (2022) Explaining image classifications with near misses, near hits and prototypes. International conference on pattern recognition and artificial intelligence (ICPRAI). Springer, Cham, pp 419\u2013430"},{"issue":"2","key":"786_CR38","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","volume":"3","author":"A Holzinger","year":"2016","unstructured":"Holzinger A (2016) Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inf 3(2):119\u2013131","journal-title":"Brain Inf"},{"issue":"1","key":"786_CR39","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1007\/s00287-015-0941-6","volume":"39","author":"A Holzinger","year":"2016","unstructured":"Holzinger A (2016) Interactive machine learning (IML). Informatik-Spektrum 39(1):64\u201368","journal-title":"Informatik-Spektrum"},{"issue":"4","key":"786_CR40","volume":"9","author":"A Holzinger","year":"2019","unstructured":"Holzinger A, Langs G, Denk H, Zatloukal K, M\u00fcller H (2019) Causability and explainability of artificial intelligence in medicine. Wiley Interdiscip Rev 9(4):e1312","journal-title":"Wiley Interdiscip Rev"},{"issue":"2","key":"786_CR41","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3446369","volume":"64","author":"S Kambhampati","year":"2021","unstructured":"Kambhampati S (2021) Polanyi\u2019s revenge and AI\u2019s new romance with tacit knowledge. Commun ACM 64(2):31\u201332","journal-title":"Commun ACM"},{"key":"786_CR42","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1146\/annurev.psych.57.102904.190100","volume":"57","author":"FC Keil","year":"2006","unstructured":"Keil FC (2006) Explanation and understanding. Annu Rev Psychol 57:227\u2013254","journal-title":"Annu Rev Psychol"},{"key":"786_CR43","first-page":"29","volume-title":"Advances in neural information processing systems","author":"B Kim","year":"2016","unstructured":"Kim B, Khanna R, Koyejo OO (2016) Examples are not enough, learn to criticize! Criticism for interpretability. Advances in neural information processing systems. Springer, Cham, p 29"},{"key":"786_CR44","first-page":"551","volume":"37","author":"K Keiichi","year":"2020","unstructured":"Keiichi K (2020) Learning by teaching face-to-face: the contributions of preparing-to-teach, initial-explanation, and interaction phases. Eur J Psychol Educ 37:551\u2013566","journal-title":"Eur J Psychol Educ"},{"issue":"1","key":"786_CR45","first-page":"25","volume":"12","author":"B Kozok","year":"1998","unstructured":"Kozok B (1998) Erkl\u00e4rungsfunktionalit\u00e4t von expertensystemen: Anwendungs- und forschungsbedarf. K\u00fcnstliche Intell 12(1):25\u201333","journal-title":"K\u00fcnstliche Intell"},{"key":"786_CR46","unstructured":"Kuehl N, Jodie L, Christian M (2020) Do you comply with AI? Personalized explanations of learning algorithms and their impact on employees\u2019 compliance behavior. arXiv:2002.0877"},{"key":"786_CR47","doi-asserted-by":"crossref","unstructured":"Kulesza T, Margaret B, Weng-Keen W, Simone S (2015) Principles of explanatory debugging to personalize interactive machine learning. In: Proceedings of 20th International Conference on Intelligent User Interfaces, pp 126\u2013137","DOI":"10.1145\/2678025.2701399"},{"issue":"7","key":"786_CR48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1108\/INTR-08-2021-0600","volume":"32","author":"S Laato","year":"2021","unstructured":"Laato S, Tiainen M, Najmul Islam AKM, M\u00e4ntym\u00e4ki M (2021) How to explain AI systems to end users: a systematic literature review and research agenda. Internet Res 32(7):1\u201331","journal-title":"Internet Res"},{"key":"786_CR49","doi-asserted-by":"crossref","unstructured":"Lakkaraju H, Stephen\u00a0HB, Jure L (2016) Interpretable decision sets: a joint framework for description and prediction. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1675\u20131684","DOI":"10.1145\/2939672.2939874"},{"key":"786_CR50","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103473","volume":"296","author":"M Langer","year":"2021","unstructured":"Langer M, Oster D, Speith T, Hermanns H, K\u00e4stner L, Schmidt E, Sesing A, Baum K (2021) What do we want from Explainable Artificial Intelligence (XAI)? A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Artif Intell 296:103473","journal-title":"Artif Intell"},{"key":"786_CR51","doi-asserted-by":"publisher","first-page":"917565","DOI":"10.3389\/frai.2022.917565","volume":"5","author":"C Langley","year":"2022","unstructured":"Langley C, Bogdan-Ionut C, Fabio C, Barbara JS (2022) Editorial: theory of mind in humans and in machines. Front Artif Intell 5:917565","journal-title":"Front Artif Intell"},{"issue":"1","key":"786_CR52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-019-08987-4","volume":"10","author":"S Lapuschkin","year":"2019","unstructured":"Lapuschkin S, W\u00e4ldchen S, Binder A, Montavon G, Samek W, M\u00fcller K-R (2019) Unmasking Clever Hans predictors and assessing what machines really learn. Nat Commun 10(1):1\u20138","journal-title":"Nat Commun"},{"issue":"3","key":"786_CR53","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1016\/j.cogpsych.2006.09.006","volume":"55","author":"T Lombrozo","year":"2007","unstructured":"Lombrozo T (2007) Simplicity and probability in causal explanation. Cogn Psychol 55(3):232\u2013257","journal-title":"Cogn Psychol"},{"key":"786_CR54","first-page":"107","volume-title":"Proceedings of the third European working session on learning","author":"D Michie","year":"1988","unstructured":"Michie D (1988) Machine learning in the next five years. Proceedings of the third European working session on learning. Pitman, Cham, pp 107\u2013122"},{"key":"786_CR55","unstructured":"Martijn M, Toon W, Katrien V (2021) Your eyes explain everything: exploring the use of eye tracking to provide explanations on-the-fly. In: Proceedings of 8th Joint Workshop on Interfaces and Human Decision Making for Recommender Systems co-located with 15th ACM Conference on Recommender Systems (RecSys 2021), volume 2948"},{"key":"786_CR56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: Insights from the social sciences. Artif Intell 267:1\u201338","journal-title":"Artif Intell"},{"issue":"3\u20134","key":"786_CR57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3387166","volume":"11","author":"S Mohseni","year":"2021","unstructured":"Mohseni S, Zarei N, Ragan ED (2021) A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Trans Interact Intell Syst 11(3\u20134):1\u201345","journal-title":"ACM Trans Interact Intell Syst"},{"key":"786_CR58","unstructured":"Molnar C (2020) Interpretable machine learning. Lulu.com"},{"issue":"7","key":"786_CR59","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1007\/s10994-018-5707-3","volume":"107","author":"SH Muggleton","year":"2018","unstructured":"Muggleton SH, Schmid U, Zeller C, Tamaddoni-Nezhad A, Besold T (2018) Ultra-strong machine learning: comprehensibility of programs learned with ILP. Mach Learn 107(7):1119\u20131140","journal-title":"Mach Learn"},{"key":"786_CR60","first-page":"12580","volume-title":"Thirty-sixth AAAI conference on artificial intelligence, AAAI 2022","author":"D Mueller","year":"2022","unstructured":"Mueller D, Michael M, Stephan S, Ute S (2022) An interactive explanatory AI system for industrial quality control. Thirty-sixth AAAI conference on artificial intelligence, AAAI 2022, thirty-fourth conference on innovative applications of artificial intelligence, IAAI 2022, the twelveth symposium on educational advances in artificial intelligence, EAAI 2022 virtual event. AAAI Press, London, pp 12580\u201312586"},{"key":"786_CR61","first-page":"55","volume-title":"Explainable AI, LNAI 11700","author":"A Nguyen","year":"2019","unstructured":"Nguyen A, Yosinski J, Clune J (2019) Understanding neural networks via feature visualization: a survey. In: Samek W et al (eds) Explainable AI, LNAI 11700. Springer, Cham, pp 55\u201376"},{"issue":"4","key":"786_CR62","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1007\/BF00168958","volume":"4","author":"HS Nwana","year":"1990","unstructured":"Nwana HS (1990) Intelligent tutoring systems: an overview. Artif Intell Rev 4(4):251\u2013277","journal-title":"Artif Intell Rev"},{"issue":"1","key":"786_CR63","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1177\/1461445616683596","volume":"19","author":"U Quasthoff","year":"2017","unstructured":"Quasthoff U, Heller V, Morek M (2017) On the sequential organization and genre-orientation of discourse units in interaction: an analytic framework. Discourse Stud 19(1):84\u2013110","journal-title":"Discourse Stud"},{"key":"786_CR64","first-page":"148","volume-title":"KI 2020: Advances in artificial intelligence \u2013 43rd German conference on AI, Bamberg, Germany, september 21-25, 2020, proceedings, volume 12325 of lecture notes in computer science","author":"J Rabold","year":"2020","unstructured":"Rabold J, Gesina S, Ute S (2020) Expressive explanations of DNNs by combining concept analysis with ILP. In: Ute S, Franziska K, Diedrich W (eds) KI 2020: Advances in artificial intelligence \u2013 43rd German conference on AI, Bamberg, Germany, september 21-25, 2020, proceedings, volume 12325 of lecture notes in computer science. Springer, Cham, pp 148\u2013162"},{"key":"786_CR65","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1007\/978-3-319-99960-9_7","volume-title":"International conference on inductive logic programming","author":"J Rabold","year":"2018","unstructured":"Rabold J, Siebers M, Schmid U (2018) Explaining black-box classifiers with ILP-empowering LIME with Aleph to approximate non-linear decisions with relational rules. International conference on inductive logic programming. Springer, Cham, pp 105\u2013117"},{"issue":"5","key":"786_CR66","doi-asserted-by":"publisher","first-page":"1799","DOI":"10.1007\/s10994-021-06048-w","volume":"111","author":"J Rabold","year":"2022","unstructured":"Rabold J, Siebers M, Schmid U (2022) Generating contrastive explanations for inductive logic programming based on a near miss approach. Mach Learn 111(5):1799\u20131820","journal-title":"Mach Learn"},{"key":"786_CR67","doi-asserted-by":"crossref","unstructured":"Ribeiro\u00a0TM, Sameer S, Carlos G (2016) Why should I trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"786_CR68","doi-asserted-by":"publisher","first-page":"767","DOI":"10.1207\/s15516709cog2606_3","volume":"26","author":"LJ Rips","year":"2002","unstructured":"Rips LJ (2002) Circular reasoning. Cogn Sci 26:767\u2013795","journal-title":"Cogn Sci"},{"issue":"3","key":"786_CR69","doi-asserted-by":"publisher","first-page":"717","DOI":"10.1109\/TCDS.2020.3044366","volume":"13","author":"KJ Rohlfing","year":"2021","unstructured":"Rohlfing KJ, Cimiano P, Scharlau I, Matzner T, Buhl HM, Buschmeier H, Esposito E, Grimminger A, Hammer B, H\u00e4b-Umbach R, Horwath I, H\u00fcllermeier E, Kern F, Kopp S, Thommes K, Ngomo A-CN, Schulte C, Wachsmuth H, Wagner P, Wrede B (2021) Explanation as a social practice: toward a conceptual framework for the social design of AI systems. IEEE Trans Cogn Dev Syst 13(3):717\u2013728","journal-title":"IEEE Trans Cogn Dev Syst"},{"issue":"2","key":"786_CR70","first-page":"4","volume":"22","author":"T Roth-Berghofer","year":"2008","unstructured":"Roth-Berghofer T, Richter MM (2008) Gasteditorial erkl\u00e4rungen. K\u00fcnstliche Intell 22(2):4","journal-title":"K\u00fcnstliche Intell"},{"key":"786_CR71","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1207\/s15516709cog2605_1","volume":"26","author":"LR Rozenblit","year":"2002","unstructured":"Rozenblit LR, Keil FC (2002) The misunderstood limits of folk science: an illusion of explanatory depth. Cogn Sci 26:521\u2013562","journal-title":"Cogn Sci"},{"issue":"5","key":"786_CR72","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","volume":"1","author":"C Rudin","year":"2019","unstructured":"Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 1(5):206\u2013215","journal-title":"Nat Mach Intell"},{"key":"786_CR73","volume-title":"Human compatible: artificial intelligence and the problem of control","author":"R Stuart","year":"2019","unstructured":"Stuart R (2019) Human compatible: artificial intelligence and the problem of control. Penguin, London"},{"key":"786_CR74","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-28954-6","volume-title":"Explainable AI: interpreting, explaining and visualizing deep learning","author":"W Samek","year":"2019","unstructured":"Samek W, Montavon G, Vedaldi A, Hansen LK, M\u00fcller K-R (2019) Explainable AI: interpreting, explaining and visualizing deep learning, vol 11700. Springer Nature, Cham"},{"key":"786_CR75","first-page":"147","volume-title":"Machine learning and knowledge discovery in databases \u2013 international workshops of ECML PKDD 2019, W\u00fcrzburg, Germany, september 16-20, 2019, proceedings, part i, volume 1167 of communications in computer and information science","author":"L Schallner","year":"2019","unstructured":"Schallner L, Rabold J, Scholz O, Schmid U (2019) Effect of superpixel aggregation on explanations in LIME: a case study with biological data. In: Cellier P, Driessens K (eds) Machine learning and knowledge discovery in databases \u2013 international workshops of ECML PKDD 2019, W\u00fcrzburg, Germany, september 16-20, 2019, proceedings, part i, volume 1167 of communications in computer and information science. Springer, Cham, pp 147\u2013158"},{"key":"786_CR76","first-page":"4","volume-title":"Proceedings of 7th workshop on dynamics of knowledge and belief (DKB-2018)","author":"U Schmid","year":"2018","unstructured":"Schmid U (2018) Inductive programming as approach to comprehensible machine learning. In: Beierle C, Kern-Isberner G, Ragni M, Stolzenburg F, Thimm M (eds) Proceedings of 7th workshop on dynamics of knowledge and belief (DKB-2018) and the 6th workshop KI and kognition (KIK-2018) co-located with 41st German conference on artificial intelligence (KI 2018), Berlin, Germany, September 25, 2018, volume 2194 of CEUR workshop proceedings. CEUR-WS.org, Berlin, pp 4\u201312"},{"key":"786_CR77","first-page":"337","volume-title":"Human-like machine intelligence","author":"U Schmid","year":"2021","unstructured":"Schmid U (2021) Interactive learning with mutual explanations in relational domains. In: Muggleton S, Charter N (eds) Human-like machine intelligence. Oxford University Press, Oxford, pp 337\u2013353"},{"issue":"2","key":"786_CR78","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1007\/s13218-020-00633-2","volume":"34","author":"U Schmid","year":"2020","unstructured":"Schmid U, Finzel B (2020) Mutual explanations for cooperative decision making in medicine. KI-K\u00fcnstliche Intell 34(2):227\u2013233","journal-title":"KI-K\u00fcnstliche Intell"},{"key":"786_CR79","first-page":"91","volume-title":"Jahrestagung der Gesellschaft f\u00fcr Informatik, INFORMATIK 2020 - back to the future, Karlsruhe, Germany, 28. september - 2. Oktober 2020, volume P-307 of LNI","author":"U Schmid","year":"2020","unstructured":"Schmid U, Tresp V, Bethge M, Kersting K, Stiefelhagen R (2020) K\u00fcnstliche intelligenz - die dritte welle. In: Reussner RH, Koziolek A, Heinrich R (eds) Jahrestagung der Gesellschaft f\u00fcr Informatik, INFORMATIK 2020 - back to the future, Karlsruhe, Germany, 28. september - 2. Oktober 2020, volume P-307 of LNI. GI, Karlsruhe, pp 91\u201395"},{"issue":"1","key":"786_CR80","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1007\/s13218-018-0565-5","volume":"33","author":"M Siebers","year":"2019","unstructured":"Siebers M, Schmid U (2019) Please delete that! Why should I? Explaining learned irrelevance classifications of digital objects. KI-K\u00fcnstliche Intell 33(1):35\u201344","journal-title":"KI-K\u00fcnstliche Intell"},{"key":"786_CR81","doi-asserted-by":"crossref","unstructured":"Sokol K, Flach P (2020) Explainability fact sheets: a framework for systematic assessment of explainable approaches. In: Proceedings of 2020 Conference on Fairness, Accountability, and Transparency, pp 56\u201467","DOI":"10.1145\/3351095.3372870"},{"issue":"2","key":"786_CR82","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s13218-020-00662-x","volume":"34","author":"S Teso","year":"2020","unstructured":"Teso S, Hinz O (2020) Challenges in interactive machine learning. KI-K\u00fcnstliche Intell 34(2):127\u2013130","journal-title":"KI-K\u00fcnstliche Intell"},{"key":"786_CR83","doi-asserted-by":"crossref","unstructured":"Teso S, Kristian K (2019) Explanatory interactive machine learning. In: Proceedings of 2019 AAAI\/ACM Conf on AI, Ethics, and Society, pp 239\u2013245","DOI":"10.1145\/3306618.3314293"},{"key":"786_CR84","unstructured":"Thaler A, Ute S (2022) Enriching intelligent tutoring systems (ITS) with methods of explainable artificial intelligence (XA) \u2013 a research proposal. Reflection on intelligent systems: towards a cross-disciplinary definition, Stuttgart"},{"key":"786_CR85","first-page":"1705","volume":"43","author":"AM Thaler","year":"2021","unstructured":"Thaler AM, Ute S (2021) Explaining machine learned relational concepts in visual domains-effects of perceived accuracy on joint performance and trust. Proc Annu Meet Cogn Sci Soc 43:1705\u20131711","journal-title":"Proc Annu Meet Cogn Sci Soc"},{"key":"786_CR86","doi-asserted-by":"crossref","unstructured":"Tintarev N, Judith M (2007) A survey of explanations in recommender systems. In: 2007 IEEE 23rd International Conference on Data Engineering Workshop, IEEE, pp 801\u2013810","DOI":"10.1109\/ICDEW.2007.4401070"},{"issue":"11","key":"786_CR87","doi-asserted-by":"publisher","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","volume":"32","author":"E Tjoa","year":"2021","unstructured":"Tjoa E, Guan C (2021) A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans Neural Netw Learn Syst 32(11):4793\u20134813","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"issue":"4","key":"786_CR88","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1080\/00461520.2011.611369","volume":"46","author":"K VanLehn","year":"2011","unstructured":"VanLehn K (2011) The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol 46(4):197\u2013221","journal-title":"Educ Psychol"},{"key":"786_CR89","first-page":"841","volume":"31","author":"S Wachter","year":"2017","unstructured":"Wachter S, Mittelstadt B, Russell C (2017) Counterfactual explanations without opening the black box: automated decisions and the GDPR. Harv JL Tech 31:841","journal-title":"Harv JL Tech"},{"key":"786_CR90","first-page":"1","volume-title":"Designing theory-driven user-centric explainable AI","author":"D Wang","year":"2019","unstructured":"Wang D, Yang Q, Abdul A, Lim BY (2019) Designing theory-driven user-centric explainable AI. Association for Computing Machinery, New York, pp 1\u201315"},{"key":"786_CR91","doi-asserted-by":"crossref","unstructured":"Wang X, Shengcheng Y, Hui Z, Michael L, Katia S (2019) Verbal explanations for deep reinforcement learning neural networks with attention on extracted features. In: 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), IEEE, pp 1\u20137","DOI":"10.1109\/RO-MAN46459.2019.8956301"},{"key":"786_CR92","doi-asserted-by":"crossref","unstructured":"Weitz K (2021) Towards human-centered AI: psychological concepts as foundation for empirical XAI research. it - Information Technology","DOI":"10.1515\/itit-2021-0047"},{"issue":"2","key":"786_CR93","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1111\/j.1469-7610.1976.tb00381.x","volume":"17","author":"D Wood","year":"1976","unstructured":"Wood D, Bruner JS, Ross G (1976) The role of tutoring in problem solving. J Child Psychol Psychiatry 17(2):89\u2013100","journal-title":"J Child Psychol Psychiatry"},{"key":"786_CR94","first-page":"10965","volume-title":"Advances in neural information processing systems 32: annual conference on neural information processing systems","author":"C-K Yeh","year":"2019","unstructured":"Yeh C-K, Hsieh C-Y, Suggala AS, Inouye DI, Ravikumar P (2019) On the (in)fidelity and sensitivity of explanations. In: Wallach HM, Larochelle H, Beygelzimer A, d\u2019Alch\u00e9-Buc F, Fox EB, Garnett R (eds) Advances in neural information processing systems 32: annual conference on neural information processing systems. NeurIPS, Vancouver, pp 10965\u201310976"},{"key":"786_CR95","series-title":"CEUR workshop proceedings","first-page":"108","volume-title":"Workshops proceedings for the 24th international conference on case-based reasoning (ICCBR 2016)","author":"C Zeller","year":"2016","unstructured":"Zeller C, Ute S (2016) Automatic generation of analogous problems to help resolving misconceptions in an intelligent tutor system for written subtraction. In: Coman A, Kapetanakis S (eds) Workshops proceedings for the 24th international conference on case-based reasoning (ICCBR 2016), vol 1815. CEUR workshop proceedings. CEUR-WS.org, Berlin, pp 108\u2013117"},{"issue":"5","key":"786_CR96","doi-asserted-by":"publisher","first-page":"593","DOI":"10.3390\/electronics10050593","volume":"10","author":"J Zhou","year":"2021","unstructured":"Zhou J, Gandomi AH, Chen F, Holzinger A (2021) Evaluating the quality of machine learning explanations: a survey on methods and metrics. Electronics 10(5):593","journal-title":"Electronics"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00786-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13218-022-00786-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00786-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T18:06:12Z","timestamp":1672164372000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13218-022-00786-2"}},"subtitle":["An Interdisciplinary Perspective"],"short-title":[],"issued":{"date-parts":[[2022,12]]},"references-count":96,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["786"],"URL":"https:\/\/doi.org\/10.1007\/s13218-022-00786-2","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12]]},"assertion":[{"value":"28 October 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}