{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T17:27:50Z","timestamp":1790702870679,"version":"4.1.0"},"reference-count":50,"publisher":"SAGE Publications","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IA"],"published-print":{"date-parts":[[2020,9,17]]},"abstract":"<jats:p>\u00a0The more intelligent systems based on sub-symbolic techniques pervade our everyday lives, the less human can understand them. This is why symbolic approaches are getting more and more attention in the general effort to make AI interpretable, explainable, and trustable. Understanding the current state of the art of AI techniques integrating symbolic and sub-symbolic approaches is then of paramount importance, nowadays\u2014in particular in the XAI perspective. This is why this paper provides an overview of the main symbolic\/sub-symbolic integration techniques, focussing in particular on those targeting explainable AI systems.<\/jats:p>","DOI":"10.3233\/ia-190036","type":"journal-article","created":{"date-parts":[[2020,9,25]],"date-time":"2020-09-25T06:13:24Z","timestamp":1601014404000},"page":"7-32","source":"Crossref","is-referenced-by-count":64,"title":["On the integration of symbolic and sub-symbolic techniques for XAI: A survey"],"prefix":"10.1177","volume":"14","author":[{"given":"Roberta","family":"Calegari","sequence":"first","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria (DISI), \u2013Universit\u00e0 di Bologna, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giovanni","family":"Ciatto","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria (DISI), \u2013Universit\u00e0 di Bologna, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Omicini","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria (DISI), \u2013Universit\u00e0 di Bologna, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","reference":[{"key":"10.3233\/IA-190036_ref1","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1007\/978-3-319-90869-4_6","author":"Helbing","year":"2019","journal-title":"Towards Digital Enlightenment"},{"key":"10.3233\/IA-190036_ref2","doi-asserted-by":"crossref","DOI":"10.4324\/9781315387185","volume-title":"The Culture of AI: Everyday Life and the Digital Revolution","author":"Elliott","year":"2019"},{"issue":"4","key":"10.3233\/IA-190036_ref3","doi-asserted-by":"publisher","first-page":"2432","DOI":"10.1109\/COMST.2017.2707140","article-title":"State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow\u2019s Intelligent Network Traffic Control Systems","volume":"19","author":"Fadlullah","year":"2017","journal-title":"IEEE Communications Surveys Tutorials"},{"issue":"3","key":"10.3233\/IA-190036_ref4","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The Mythos of Model Interpretability","volume":"16","author":"Lipton","year":"2018","journal-title":"Queue"},{"issue":"2","key":"10.3233\/IA-190036_ref5","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1609\/aimag.v40i2.2850","article-title":"DARPA\u2019s Explainable Artificial Intelligence Program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Magazine"},{"issue":"December 2019","key":"10.3233\/IA-190036_ref6","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Barredo Arrieta","year":"2020","journal-title":"Information Fusion"},{"key":"10.3233\/IA-190036_ref7","doi-asserted-by":"crossref","unstructured":"Goertzel B. , Perception Processing for General Intelligence: Bridging the Symbolic\/Subsymbolic Gap, in: Artificial General Intelligence, J. Bach, B. Goertzel and M. Ikl\u00e9, eds, Springer Berlin Heidelberg, 2012, pp. 79\u201388. ISBN ISBN 978-3-642-35506-6.","DOI":"10.1007\/978-3-642-35506-6_9"},{"key":"10.3233\/IA-190036_ref8","doi-asserted-by":"publisher","DOI":"10.3390\/bdcc2030023"},{"key":"10.3233\/IA-190036_ref9","unstructured":"Calegari R. , Ciatto G. , Dellaluce J. , Omicini A. , Interpretable Narrative Explanation for ML Predictors with LP: A Case Study for XAI, in: WOA 2019 \u2019 20th Workshop \u201cFrom Objects to Agents\u201d, F. Bergenti and S. Monica, eds, CEUR Workshop Proceedings, Vol. 2404, Sun SITE Central Europe, RWTH Aachen University, 2019, pp. 105\u2013112. http:\/\/ceur-ws.org\/Vol-2404\/paper16.pdf"},{"key":"10.3233\/IA-190036_ref12","doi-asserted-by":"publisher","DOI":"10.1145\/3236009"},{"key":"10.3233\/IA-190036_ref13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-25932-9_1"},{"key":"10.3233\/IA-190036_ref18","first-page":"334","author":"Serafini","year":"2016","journal-title":"Conference of the Italian Association for Artificial Intelligence"},{"key":"10.3233\/IA-190036_ref19","doi-asserted-by":"publisher","DOI":"10.1145\/3019612.3019642"},{"key":"10.3233\/IA-190036_ref21","unstructured":"Marra G. , Giannini F. , Diligenti M. and Gori M. , LYRICS: a General Interface Layer to Integrate AI and Deep Learning, arXiv preprint arXiv:1903.07534, (2019)."},{"issue":"1\u20132","key":"10.3233\/IA-190036_ref22","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s10994-006-5833-1","article-title":"Markov logic networks","volume":"62","author":"Richardson","year":"2006","journal-title":"Machine Learning"},{"issue":"1","key":"10.3233\/IA-190036_ref23","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1007\/s10994-013-5392-1","article-title":"Fast relational learning using bottom clause propositionalization with artificial neural networks","volume":"94","author":"Fran\u00e7a","year":"2014","journal-title":"Machine Learning"},{"key":"10.3233\/IA-190036_ref24","doi-asserted-by":"publisher","DOI":"10.1613\/jair.5714"},{"key":"10.3233\/IA-190036_ref25","first-page":"3788","author":"Rocktaschel","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10.3233\/IA-190036_ref27","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1613\/jair.1.11203","article-title":"Lifted Relational Neural Networks: Efficient Learning of Latent Relational Structures","volume":"62","author":"Sourek","year":"2018","journal-title":"Journal of Artificial Intelligence Research"},{"key":"10.3233\/IA-190036_ref28","doi-asserted-by":"publisher","DOI":"10.1109\/ICNN.1988.23855"},{"issue":"2","key":"10.3233\/IA-190036_ref29","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/s11063-011-9207-8","article-title":"Reverse engineering the neural networks for rule extraction in classification problems","volume":"35","author":"Augasta","year":"2012","journal-title":"Neural Processing Letters"},{"issue":"11","key":"10.3233\/IA-190036_ref30","doi-asserted-by":"publisher","first-page":"2664","DOI":"10.1109\/TNNLS.2015.2389037","article-title":"Active Learning-Based Pedagogical Rule Extraction","volume":"26","author":"de Fortuny","year":"2015","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"key":"10.3233\/IA-190036_ref31","first-page":"969","author":"McMillan","year":"1992","journal-title":"Advances in neural information processing systems"},{"key":"10.3233\/IA-190036_ref32","first-page":"977","author":"Towell","year":"1992","journal-title":"Advances in neural information processing systems"},{"key":"10.3233\/IA-190036_ref33","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1109\/ICNN.1993.298658","author":"Giles","year":"1993","journal-title":"IEEE International Conference on Neural Networks, IEEE"},{"issue":"8","key":"10.3233\/IA-190036_ref34","doi-asserted-by":"crossref","first-page":"1114","DOI":"10.1109\/21.299696","article-title":"Rule generation from neural networks","volume":"24","author":"Fu","year":"1994","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"10.3233\/IA-190036_ref35","first-page":"505","author":"Thrun","year":"1995","journal-title":"Advances in neural information processing systems"},{"issue":"1","key":"10.3233\/IA-190036_ref36","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1162\/neco.1997.9.1.205","article-title":"Extracting Rules from Neural Networks by Pruning and Hidden-Unit Splitting","volume":"9","author":"Setiono","year":"1997","journal-title":"Neural Computation"},{"key":"10.3233\/IA-190036_ref37","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-75390-2_5"},{"key":"10.3233\/IA-190036_ref38","first-page":"24","author":"Craven","year":"1996","journal-title":"Advances in neural information processing systems"},{"key":"10.3233\/IA-190036_ref39","doi-asserted-by":"crossref","unstructured":"Krishnan R. , Sivakumar G. and Bhattacharya P. , Extracting decision trees from trained neural networks, Pattern recognition 32(12) (1999).","DOI":"10.1016\/S0031-3203(98)00181-2"},{"issue":"3","key":"10.3233\/IA-190036_ref40","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1109\/TITB.2006.880553","article-title":"Confident interpretation of Bayesian decision tree ensembles for clinical applications","volume":"11","author":"Schetinin","year":"2007","journal-title":"IEEE Transactions on Information Technology in Biomedicine"},{"key":"10.3233\/IA-190036_ref41","unstructured":"Wang Q. , Wang B. and Guo L. , Knowledge base completion using embeddings and rules, in: Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015."},{"key":"10.3233\/IA-190036_ref43","first-page":"1119","author":"Rockt\u00e4schel","year":"2015","journal-title":"Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"},{"key":"10.3233\/IA-190036_ref44","doi-asserted-by":"crossref","first-page":"192","DOI":"10.18653\/v1\/D16-1019","author":"Guo","year":"2016","journal-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"},{"key":"10.3233\/IA-190036_ref45","unstructured":"Goodwin T.R. and Demner-Fushman D. , Bridging the Knowledge Gap: Enhancing Question Answering withWorld and Domain Knowledge, arXiv preprint arXiv:1910.07429 (2019)."},{"key":"10.3233\/IA-190036_ref47","doi-asserted-by":"crossref","unstructured":"De Raedt L. , Logical and relational learning, Springer Science & Business Media, 2008.","DOI":"10.1007\/978-3-540-68856-3"},{"key":"10.3233\/IA-190036_ref48","first-page":"109:1","article-title":"Hinge-Loss Markov Random Fields and Probabilistic Soft Logic","volume":"18","author":"Bach","year":"2017","journal-title":"Journal of Machine Learning Research"},{"key":"10.3233\/IA-190036_ref49","first-page":"926","author":"Socher","year":"2013","journal-title":"Advances in neural information processing systems"},{"key":"10.3233\/IA-190036_ref50","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511801129","volume-title":"An introduction to many-valued and fuzzy logic: semantics, algebras, and derivation systems","author":"Bergmann","year":"2008"},{"key":"10.3233\/IA-190036_ref51","first-page":"1596","author":"Donadello","year":"2017","journal-title":"Proceedings of the 26th International Joint Conference on Artificial Intelligence"},{"issue":"3","key":"10.3233\/IA-190036_ref52","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1609\/aimag.v29i3.2157","article-title":"Collective Classification in Network Data","volume":"29","author":"Sen","year":"2008","journal-title":"AI Magazine"},{"key":"10.3233\/IA-190036_ref53","first-page":"5116","author":"Belle","year":"2017","journal-title":"IJCAI"},{"issue":"12","key":"10.3233\/IA-190036_ref54","doi-asserted-by":"publisher","first-page":"1794","DOI":"10.14778\/3352063.3352068","article-title":"SystemER: a human-inthe-loop system for explainable entity resolution","volume":"12","author":"Qian","year":"2019","journal-title":"Proceedings of the VLDB Endowment"},{"key":"10.3233\/IA-190036_ref55","unstructured":"Kate R.J. and Mooney R.J. , RJ: Probabilistic abduction using Markov logic networks, in: IJCAI \u201909 Workshop on Plan, Activity, and Intent Recognition, 2009."},{"key":"10.3233\/IA-190036_ref57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/EDOC.2016.7579386","author":"Schoenfisch","year":"2016","journal-title":"2016 IEEE 20th International Enterprise Distributed Object Computing Conference (EDOC)"},{"issue":"1\u20132","key":"10.3233\/IA-190036_ref58","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/S0004-3702(00)00077-1","article-title":"Symbolic knowledge extraction from trained neural networks: A sound approach","volume":"125","author":"d\u2019Avila Garcez","year":"2001","journal-title":"Artificial Intelligence"},{"issue":"6","key":"10.3233\/IA-190036_ref59","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1016\/0950-7051(96)81920-4","article-title":"Survey and Critique of Techniques for Extracting Rules from Trained Artificial Neural Networks","volume":"8","author":"Andrews","year":"1995","journal-title":"Knowledge-Based Systems"},{"issue":"1","key":"10.3233\/IA-190036_ref60","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/j.neucom.2010.02.016","article-title":"Rule extraction from support vector machines: A review","volume":"74","author":"Barakat","year":"2010","journal-title":"Neurocomputing"},{"issue":"3","key":"10.3233\/IA-190036_ref61","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3390\/info11030167","article-title":"Logic-based Technologies for Intelligent Systems: State of the Art and Perspectives","volume":"11","author":"Calegari","year":"2020","journal-title":"Information"}],"container-title":["Intelligenza Artificiale"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/IA-190036","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:51:38Z","timestamp":1777459898000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.medra.org\/servlet\/aliasResolver?alias=iospress&doi=10.3233\/IA-190036"}},"subtitle":[],"editor":[{"given":"Matteo","family":"Baldoni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]},{"given":"Federico","family":"Bergenti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]},{"given":"Stefania","family":"Monica","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]},{"given":"Giuseppe","family":"Vizzari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2020,9,17]]},"references-count":50,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.3233\/ia-190036","relation":{},"ISSN":["1724-8035","2211-0097"],"issn-type":[{"value":"1724-8035","type":"print"},{"value":"2211-0097","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,17]]}}}