{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T11:29:56Z","timestamp":1779103796218,"version":"3.51.4"},"reference-count":53,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,2,22]],"date-time":"2020-02-22T00:00:00Z","timestamp":1582329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Deep learning models contributed to reaching unprecedented results in prediction and classification tasks of Artificial Intelligence (AI) systems. However, alongside this notable progress, they do not provide human-understandable insights on how a specific result was achieved. In contexts where the impact of AI on human life is relevant (e.g., recruitment tools, medical diagnoses, etc.), explainability is not only a desirable property, but it is -or, in some cases, it will be soon-a legal requirement. Most of the available approaches to implement eXplainable Artificial Intelligence (XAI) focus on technical solutions usable only by experts able to manipulate the recursive mathematical functions in deep learning algorithms. A complementary approach is represented by symbolic AI, where symbols are elements of a lingua franca between humans and deep learning. In this context, Knowledge Graphs (KGs) and their underlying semantic technologies are the modern implementation of symbolic AI\u2014while being less flexible and robust to noise compared to deep learning models, KGs are natively developed to be explainable. In this paper, we review the main XAI approaches existing in the literature, underlying their strengths and limitations, and we propose neural-symbolic integration as a cornerstone to design an AI which is closer to non-insiders comprehension. Within such a general direction, we identify three specific challenges for future research\u2014knowledge matching, cross-disciplinary explanations and interactive explanations.<\/jats:p>","DOI":"10.3390\/info11020122","type":"journal-article","created":{"date-parts":[[2020,2,24]],"date-time":"2020-02-24T03:33:43Z","timestamp":1582515223000},"page":"122","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":90,"title":["On the Integration of Knowledge Graphs into Deep Learning Models for a More Comprehensible AI\u2014Three Challenges for Future Research"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3866-5177","authenticated-orcid":false,"given":"Giuseppe","family":"Futia","sequence":"first","affiliation":[{"name":"Nexa Center for Internet and Society, Department of Control and Computer Engineering (DAUIN), Politecnico di Torino, 10129 Turin, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2027-3308","authenticated-orcid":false,"given":"Antonio","family":"Vetr\u00f2","sequence":"additional","affiliation":[{"name":"Nexa Center for Internet and Society, Department of Control and Computer Engineering (DAUIN), Politecnico di Torino, 10129 Turin, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,22]]},"reference":[{"key":"ref_1","first-page":"865","article-title":"Traffic flow prediction with big data: A deep learning approach","volume":"16","author":"Lv","year":"2014","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_2","unstructured":"Ding, X., Zhang, Y., Liu, T., and Duan, J. (2015, January 25\u201331). Deep learning for event-driven stock prediction. Proceedings of the Twenty-Fourth International Joint Conference on Artificial intelligence, Buenos Aires, Argentina."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"80","DOI":"10.3389\/fenvs.2015.00080","article-title":"DeepTox: Toxicity prediction using deep learning","volume":"3","author":"Mayr","year":"2016","journal-title":"Front. Environ. Sci."},{"key":"ref_4","unstructured":"Doran, D., Schulz, S., and Besold, T.R. (2017). What does explainable AI really mean? A new conceptualization of perspectives. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1999","DOI":"10.1016\/j.eswa.2010.07.134","article-title":"Employee churn prediction","volume":"38","author":"Saradhi","year":"2011","journal-title":"Expert Syst. Appl."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kang, H.W., and Kang, H.B. (2017). Prediction of crime occurrence from multi-modal data using deep learning. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0176244"},{"key":"ref_7","unstructured":"Al Hasan, M., Chaoji, V., Salem, S., and Zaki, M. (2004, January 24). Link prediction using supervised learning. Proceedings of the SDM04: Workshop on Link Analysis, Counter-Terrorism and Security, Lake Buena Vista, FL, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1038\/s41551-018-0324-9","article-title":"An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets","volume":"3","author":"Lee","year":"2019","journal-title":"Nat. Biomed. Eng."},{"key":"ref_9","unstructured":"Ehrlinger, L., and W\u00f6\u00df, W. (2020, February 21). Towards a Definition of Knowledge Graphs. SEMANTiCS (Posters, Demos, SuCCESS) 2016. Available online: https:\/\/2016.semantics.cc\/posters-and-demos-madness."},{"key":"ref_10","unstructured":"Seeliger, A., Pfaff, M., and Krcmar, H. (2020, February 21). Semantic Web Technologies for Explainable Machine Learning Models: A Literature Review. PROFILES 2019. Available online: http:\/\/ceur-ws.org\/Vol-2465\/."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chekol, M.W., Pirr\u00f2, G., Schoenfisch, J., and Stuckenschmidt, H. (2017, January 4\u20139). Marrying uncertainty and time in knowledge graphs. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10495"},{"key":"ref_12","unstructured":"Munch, M., Dibie, J., Wuillemin, P.H., and Manfredotti, C. (2020, February 21). Interactive Causal Discovery in Knowledge Graphs. Available online: http:\/\/ceur-ws.org\/Vol-2465\/."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"907","DOI":"10.1006\/ijhc.1995.1081","article-title":"Toward principles for the design of ontologies used for knowledge sharing?","volume":"43","author":"Gruber","year":"1995","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_14","unstructured":"Dumontier, M., Callahan, A., Cruz-Toledo, J., Ansell, P., Emonet, V., Belleau, F., and Droit, A. (2014, January 21). Bio2RDF release 3: A larger connected network of linked data for the life sciences. Proceedings of the 2014 International Conference on Posters & Demonstrations Track, Riva del Garda, Italy."},{"key":"ref_15","unstructured":"Banerjee, S. (2013). A Semantic Web Based Ontology in the Financial Domain. Proceedings of World Academy of Science, Engineering and Technology, World Academy of Science, Engineering and Technology (WASET). Number 78."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"213","DOI":"10.3233\/SW-160224","article-title":"Semantic web for the legal domain: The next step","volume":"7","author":"Casanovas","year":"2016","journal-title":"Semant. Web"},{"key":"ref_17","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_18","first-page":"318","article-title":"Parallel distributed processing: Explorations in the microstructure of cognition","volume":"1","author":"Rumelhart","year":"1986","journal-title":"Learn. Intern. Represent. Error Propag."},{"key":"ref_19","unstructured":"Hinton, G.E., McClelland, J.L., and Rumelhart, D.E. (1984). Distributed Representations, Carnegie-Mellon University."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"ref_21","unstructured":"Koh, P.W., and Liang, P. (2017, January 7\u20139). Understanding black-box predictions via influence functions. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lecue, F. (2020, February 21). On the Role of Knowledge Graphs in Explainable AI. Semantic Web Journal (Forthcoming). Available online: http:\/\/www.semantic-web-journal.net\/content\/role-knowledge-graphs-explainable-ai.","DOI":"10.3233\/SW-190374"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Haugeland, J. (1989). Artificial Intelligence: The Very Idea, MIT Press.","DOI":"10.7551\/mitpress\/1170.001.0001"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-031-79432-2","article-title":"Linked data: Evolving the web into a global data space","volume":"1","author":"Heath","year":"2011","journal-title":"Synth. Lect. Semant. Web Theory Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"27","DOI":"10.3233\/DS-170004","article-title":"Data science and symbolic AI: Synergies, challenges and opportunities","volume":"1","author":"Hoehndorf","year":"2017","journal-title":"Data Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_27","unstructured":"Miller, T., Howe, P., and Sonenberg, L. (2017). Explainable AI: Beware of inmates running the asylum or: How I learnt to stop worrying and love the social and behavioural sciences. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mittelstadt, B., Russell, C., and Wachter, S. (2019, January 29\u201331). Explaining explanations in AI. Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA.","DOI":"10.1145\/3287560.3287574"},{"key":"ref_29","unstructured":"Hall, P., and Gill, N. (2018). An Introduction to Machine Learning Interpretability-Dataiku Version, O\u2019Reilly Media, Incorporated."},{"key":"ref_30","unstructured":"Johansson, U., K\u00f6nig, R., and Niklasson, L. (2004, January 17\u201319). The Truth is In There-Rule Extraction from Opaque Models Using Genetic Programming. Proceedings of the International Florida Artificial Intelligence Research Society Conference, FLAIRS 2004, Miami Beach, FL, USA."},{"key":"ref_31","unstructured":"Sadowski, P., Collado, J., Whiteson, D., and Baldi, P. (2015, January 13). Deep learning, dark knowledge, and dark matter. Proceedings of the NIPS 2014 Workshop on High-energy Physics and Machine Learning, Montreal, QC, Canada."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1214\/ss\/1009213726","article-title":"Statistical modeling: The two cultures (with comments and a rejoinder by the author)","volume":"16","author":"Breiman","year":"2001","journal-title":"Stat. Sci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1007\/s13347-017-0279-x","article-title":"Fair, transparent, and accountable algorithmic decision-making processes","volume":"31","author":"Lepri","year":"2018","journal-title":"Philos. Technol."},{"key":"ref_34","unstructured":"Friedler, S.A., Roy, C.D., Scheidegger, C., and Slack, D. (2019). Assessing the local interpretability of machine learning models. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Assaf, R., and Schumann, A. (2019, January 10\u201316). Explainable deep neural networks for multivariate time series predictions. Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China.","DOI":"10.24963\/ijcai.2019\/932"},{"key":"ref_36","unstructured":"Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. (2018, January 3\u20138). Sanity checks for saliency maps. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Datta, A., Sen, S., and Zick, Y. (2016, January 22\u201326). Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA.","DOI":"10.1109\/SP.2016.42"},{"key":"ref_38","unstructured":"Lipton, Z.C. (2016). The mythos of model interpretability. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should i trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Tamagnini, P., Krause, J., Dasgupta, A., and Bertini, E. (2017, January 14). Interpreting black-box classifiers using instance-level visual explanations. Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics, Chicago, IL, USA.","DOI":"10.1145\/3077257.3077260"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Adler, P., Falk, C., Friedler, S.A., Rybeck, G., Scheidegger, C., Smith, B., and Venkatasubramanian, S. (2016). Auditing black-box models by obscuring features. arXiv.","DOI":"10.1109\/ICDM.2016.0011"},{"key":"ref_42","unstructured":"Kim, B., Rudin, C., and Shah, J.A. (2014, January 8\u201313). The bayesian case model: A generative approach for case-based reasoning and prototype classification. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_43","unstructured":"Sarker, M.K., Xie, N., Doran, D., Raymer, M., and Hitzler, P. (2017). Explaining trained neural networks with semantic web technologies: First steps. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Angelov, P., and Soares, E. (2019). Towards Explainable Deep Neural Networks (xDNN). arXiv.","DOI":"10.1016\/j.neunet.2020.07.010"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Chattopadhyay, P., Elhoseiny, M., Sharma, T., Batra, D., Parikh, D., and Lee, S. (2018, January 8\u201314). Choose Your Neuron: Incorporating Domain Knowledge through Neuron-Importance. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01261-8_32"},{"key":"ref_46","unstructured":"Batet, M., Valls, A., Gibert, K., and S\u00e1nchez, D. (2010, January 20\u201322). Semantic Clustering Using Multiple Ontologies. Proceedings of the Catalan Conference on AI (CCIA), Terragona, Spain."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1145\/219717.219748","article-title":"WordNet: A lexical database for English","volume":"38","author":"Miller","year":"1995","journal-title":"Commun. ACM"},{"key":"ref_48","unstructured":"Geng, Y., Chen, J., Jimenez-Ruiz, E., and Chen, H. (2019). Human-centric transfer learning explanation via knowledge graph. arXiv."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Wilcke, X., Bloem, P., and de Boer, V. (2017). The Knowledge Graph as the Default Data Model for Machine Learning, IOS Press.","DOI":"10.3233\/DS-170007"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Sopchoke, S., Fukui, K.i., and Numao, M. (2018, January 2\u20137). Explainable cross-domain recommendations through relational learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12176"},{"key":"ref_51","unstructured":"Wang, P., Wu, Q., Shen, C., Hengel, A.V., and Dick, A. (2015). Explicit knowledge-based reasoning for visual question answering. arXiv."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"167","DOI":"10.3233\/SW-140134","article-title":"DBpedia\u2013a large-scale, multilingual knowledge base extracted from Wikipedia","volume":"6","author":"Lehmann","year":"2015","journal-title":"Semant. Web"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Liao, L., He, X., Zhao, B., Ngo, C.W., and Chua, T.S. (2018, January 22\u201326). Interpretable multimodal retrieval for fashion products. Proceedings of the 2018 ACM Multimedia Conference on Multimedia Conference, Seoul, Korea.","DOI":"10.1145\/3240508.3240646"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/2\/122\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T08:59:57Z","timestamp":1760173197000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/2\/122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,22]]},"references-count":53,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,2]]}},"alternative-id":["info11020122"],"URL":"https:\/\/doi.org\/10.3390\/info11020122","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,22]]}}}