{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T10:01:15Z","timestamp":1769940075408,"version":"3.49.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2022,7,9]],"date-time":"2022-07-09T00:00:00Z","timestamp":1657324800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,9]],"date-time":"2022-07-09T00:00:00Z","timestamp":1657324800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Otto-Friedrich-Universit\u00e4t Bamberg"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep learning methods, although effective in their assigned tasks, are mostly black-boxes with respect to their inner workings. For image classification with CNNs, there exists a variety of visual explanation methods that highlight parts of input images that were relevant for the classification result. But in many domains visual highlighting may not be expressive enough when the classification relies on complex relations within visual concepts. This paper presents an approach to enrich visual explanations with verbal local explanations, emphasizing important relational information. The proposed <jats:sc>SymMetric<\/jats:sc> algorithm combines metric learning and inductive logic programming (ILP). Labels given by a human for a small subset of important image parts are first generalized to a neighborhood of similar images using a learned distance metric. The information about labels and their spatial relations is then used to build background knowledge for ILP and ultimately to learn a first-order theory that locally explains the black-box with respect to the given image. The approach is evaluated with the Dogs vs. Cats data set demonstrating the generalization ability of metric learning and with Picasso Faces to illustrate recognition of spatial meaningful constellations of sub-concepts and creation of an expressive explanation.<\/jats:p>","DOI":"10.1007\/s13218-022-00771-9","type":"journal-article","created":{"date-parts":[[2022,7,9]],"date-time":"2022-07-09T03:27:42Z","timestamp":1657337262000},"page":"225-235","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A Neural-Symbolic Approach for Explanation Generation Based on Sub-concept Detection: An Application of Metric Learning for Low-Time-Budget Labeling"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0656-5881","authenticated-orcid":false,"given":"Johannes","family":"Rabold","sequence":"first","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,7,9]]},"reference":[{"key":"771_CR1","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE Access 6:52138\u201352160","journal-title":"IEEE Access"},{"key":"771_CR2","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garc\u00eda S, Gil-L\u00f3pez S, Molina D, Benjamins R et al (2020) Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. Inf Fusion 58:82\u2013115","journal-title":"Inf Fusion"},{"issue":"7","key":"771_CR3","doi-asserted-by":"publisher","first-page":"e013040","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach S, Binder A, Montavon G, Klauschen F, M\u00fcller K-R, Samek W (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 10(7):e013040","journal-title":"PLoS One"},{"key":"771_CR4","doi-asserted-by":"publisher","first-page":"103649","DOI":"10.1016\/j.artint.2021.103649","volume":"303","author":"S Badreddine","year":"2022","unstructured":"Badreddine S, Garcez AD, Serafini L (2022) Logic tensor networks. Artif Intell 303:103649","journal-title":"Artif Intell"},{"key":"771_CR5","doi-asserted-by":"crossref","unstructured":"Bau D, Zhou B, Khosla A, Oliva A, Torralba A (2017) Network dissection: quantifying interpretability of deep visual representations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Manhattan, pp 6541\u20136549","DOI":"10.1109\/CVPR.2017.354"},{"issue":"1","key":"771_CR6","first-page":"1","volume":"9","author":"A Bellet","year":"2015","unstructured":"Bellet A, Habrard A, Sebban M (2015) Metric learning. Synth Lect Artif Intell Mach Learn 9(1):1\u2013151","journal-title":"Synth Lect Artif Intell Mach Learn"},{"key":"771_CR7","doi-asserted-by":"crossref","unstructured":"Dai W-Z, Muggleton S, Wen J, Tamaddoni-Nezhad A, Zhou Z-H (2017) Logical vision: one-shot meta-interpretive learning from real images. In: International conference on inductive logic programming. Springer, pp 46\u201362","DOI":"10.1007\/978-3-319-78090-0_4"},{"key":"771_CR8","first-page":"366","volume":"7","author":"J Elson","year":"2007","unstructured":"Elson J, Douceur JR, Howell J, Saul J (2007) Asirra: a captcha that exploits interest-aligned manual image categorization. ACM Conf Comput Commun Secur 7:366\u2013374","journal-title":"ACM Conf Comput Commun Secur"},{"key":"771_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1613\/jair.5714","volume":"61","author":"R Evans","year":"2018","unstructured":"Evans R, Grefenstette E (2018) Learning explanatory rules from noisy data. J Artif Intell Res 61:1\u201364","journal-title":"J Artif Intell Res"},{"key":"771_CR10","doi-asserted-by":"crossref","unstructured":"Fong R, Vedaldi A (2018) Net2vec: quantifying and explaining how concepts are encoded by filters in deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Manhattan, pp 8730\u20138738","DOI":"10.1109\/CVPR.2018.00910"},{"key":"771_CR11","unstructured":"Garcez AD, Gori M, Lamb LC, Serafini L, Spranger M, Tran SN (2019) Neural-symbolic computing: an effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv:1905.06088"},{"key":"771_CR12","unstructured":"Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. http:\/\/www.deeplearningbook.org"},{"issue":"5","key":"771_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D (2018) A survey of methods for explaining black box models. ACM Comput Surv (CSUR) 51(5):1\u201342","journal-title":"ACM Comput Surv (CSUR)"},{"key":"771_CR14","doi-asserted-by":"crossref","unstructured":"Khan K, Mauro M, Leonardi R (2015) Multi-class semantic segmentation of faces. In: 2015 IEEE international conference on image processing (ICIP). IEEE, pp 827\u2013831","DOI":"10.1109\/ICIP.2015.7350915"},{"issue":"4","key":"771_CR15","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1561\/2200000019","volume":"5","author":"B Kulis","year":"2012","unstructured":"Kulis B et al (2012) Metric learning: a survey. Found Trends Mach Learn 5(4):287\u2013364","journal-title":"Found Trends Mach Learn"},{"key":"771_CR16","first-page":"3749","volume":"31","author":"R Manhaeve","year":"2018","unstructured":"Manhaeve R, Dumancic S, Kimmig A, Demeester T, De Raedt L (2018) Deepproblog: neural probabilistic logic programming. Adv Neural Inf Process Syst 31:3749\u20133759","journal-title":"Adv Neural Inf Process Syst"},{"key":"771_CR17","unstructured":"Michie D (1988) Machine learning in the next five years. In: Proceedings of the 3rd European conference on European working session on learning. Pitman Publishing, Inc, Marshfield, pp 107\u2013122"},{"key":"771_CR18","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546"},{"key":"771_CR19","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1016\/j.patcog.2016.11.008","volume":"65","author":"G Montavon","year":"2017","unstructured":"Montavon G, Lapuschkin S, Binder A, Samek W, M\u00fcller K-R (2017) Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit 65:211\u2013222","journal-title":"Pattern Recognit"},{"key":"771_CR20","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1016\/0743-1066(94)90035-3","volume":"19","author":"S Muggleton","year":"1994","unstructured":"Muggleton S, De Raedt L (1994) Inductive logic programming: theory and methods. J Logic Program 19:629\u2013679","journal-title":"J Logic Program"},{"issue":"7","key":"771_CR21","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1007\/s10994-018-5707-3","volume":"107","author":"SH Muggleton","year":"2018","unstructured":"Muggleton SH, Schmid U, Zeller C, Tamaddoni-Nezhad A, Besold T (2018) Ultra-strong machine learning: comprehensibility of programs learned with ilp. Mach Learn 107(7):1119\u20131140","journal-title":"Mach Learn"},{"key":"771_CR22","doi-asserted-by":"crossref","unstructured":"M\u00fcller D, M\u00e4rz M, Scheele S, Schmid U (2022) An interactive explanatory ai system for industrial quality control","DOI":"10.1609\/aaai.v36i11.21530"},{"key":"771_CR23","doi-asserted-by":"crossref","unstructured":"Rabold J, Siebers M, Schmid U (2018) Explaining black-box classifiers with ilp-empowering lime with aleph to approximate non-linear decisions with relational rules. In: International conference on inductive logic programming. Springer, pp 105\u2013117","DOI":"10.1007\/978-3-319-99960-9_7"},{"key":"771_CR24","doi-asserted-by":"crossref","unstructured":"Rabold J, Deininger H, Siebers M, Schmid U (2019) Enriching visual with verbal explanations for relational concepts-combining lime with aleph. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 180\u2013192","DOI":"10.1007\/978-3-030-43823-4_16"},{"key":"771_CR25","doi-asserted-by":"crossref","unstructured":"Rabold J, Schwalbe G, Schmid U (2020) Expressive explanations of DNNS by combining concept analysis with ilp. In: German conference on artificial intelligence (K\u00fcnstliche Intelligenz). Springer, pp 148\u2013162","DOI":"10.1007\/978-3-030-58285-2_11"},{"key":"771_CR26","doi-asserted-by":"publisher","unstructured":"Raedt LD, Dumancic S, Manhaeve R, Marra G (2020) From statistical relational to neuro-symbolic artificial intelligence. In: Bessiere C (ed) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI2020, pp 4943\u20134950. ijcai.org. https:\/\/doi.org\/10.24963\/ijcai.2020\/688","DOI":"10.24963\/ijcai.2020\/688"},{"key":"771_CR27","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) Why should i trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. Association for Computing Machinery, New York, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"771_CR28","doi-asserted-by":"crossref","unstructured":"Schmid U (2021) Interactive learning with mutual explanations in relational domains. Human-like machine intelligence. pp 338","DOI":"10.1093\/oso\/9780198862536.003.0017"},{"key":"771_CR29","doi-asserted-by":"crossref","unstructured":"Schmid U, Finzel B (2020)Mutual explanations for cooperative decision making in medicine. KI-K\u00fcnstliche Intelligenz. pp 1\u20137","DOI":"10.1007\/s13218-020-00633-2"},{"key":"771_CR30","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. IEEE, Piscataway, NJ, pp 618\u2013626","DOI":"10.1109\/ICCV.2017.74"},{"key":"771_CR31","unstructured":"Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint, arXiv:1409.1556"},{"key":"771_CR32","unstructured":"Srinivasan A (2007) The Aleph Manual. https:\/\/www.cs.ox.ac.uk\/activities\/programinduction\/Aleph\/aleph.html. Accessed 10 Feb 2022"},{"issue":"1","key":"771_CR33","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1145\/602382.602410","volume":"50","author":"LG Valiant","year":"2003","unstructured":"Valiant LG (2003) Three problems in computer science. J ACM (JACM) 50(1):96\u201399","journal-title":"J ACM (JACM)"},{"issue":"2","key":"771_CR34","first-page":"207","volume":"10","author":"KQ Weinberger","year":"2009","unstructured":"Weinberger KQ, Saul LK (2009) Distance metric learning for large margin nearest neighbor classification. J Mach Learn Res 10(2):207\u2013244","journal-title":"J Mach Learn Res"},{"issue":"7\u20138","key":"771_CR35","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1515\/teme-2019-0024","volume":"86","author":"K Weitz","year":"2019","unstructured":"Weitz K, Hassan T, Schmid U, Garbas J-U (2019) Deep-learned faces of pain and emotions: elucidating the differences of facial expressions with the help of explainable ai methods. tm-Technisches Messen 86(7\u20138):404\u2013412","journal-title":"tm-Technisches Messen"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00771-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13218-022-00771-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-022-00771-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T17:58:33Z","timestamp":1672163913000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13218-022-00771-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,9]]},"references-count":35,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["771"],"URL":"https:\/\/doi.org\/10.1007\/s13218-022-00771-9","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,9]]},"assertion":[{"value":"10 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 October 2022","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Missing Open Access funding information has been added in the Funding Note.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}}]}}