{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T01:01:14Z","timestamp":1775523674986,"version":"3.50.1"},"reference-count":58,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T00:00:00Z","timestamp":1663977600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["RGPIN-2020-05869"],"award-info":[{"award-number":["RGPIN-2020-05869"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Information Science"],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>In recent years, knowledge graphs (KGs) have been widely applied in various domains for different purposes. The semantic model of KGs can represent knowledge through a hierarchical structure based on classes of entities, their properties, and their relationships. The construction of large KGs can enable the integration of heterogeneous information sources and help Artificial Intelligence (AI) systems be more explainable and interpretable. This systematic review examines a selection of recent publications to understand how KGs are currently being used in eXplainable AI systems. To achieve this goal, we design a framework and divide the use of KGs into four categories: extracting features, extracting relationships, constructing KGs, and KG reasoning. We also identify where KGs are mostly used in eXplainable AI systems (pre-model, in-model, and post-model) according to the aforementioned categories. Based on our analysis, KGs have been mainly used in pre-model XAI for feature and relation extraction. They were also utilised for inference and reasoning in post-model XAI. We found several studies that leveraged KGs to explain the XAI models in the healthcare domain.<\/jats:p>","DOI":"10.1177\/01655515221112844","type":"journal-article","created":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T08:32:19Z","timestamp":1664008339000},"page":"1019-1029","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":84,"title":["Knowledge-graph-based explainable AI: A systematic review"],"prefix":"10.1177","volume":"50","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9557-0043","authenticated-orcid":false,"given":"Enayat","family":"Rajabi","sequence":"first","affiliation":[{"name":"Shannon School of Business, Cape Breton University, Canada"}]},{"given":"Kobra","family":"Etminani","sequence":"additional","affiliation":[{"name":"Center for Applied Intelligent Systems Research (CAISR), Halmstad University, Sweden"}]}],"member":"179","published-online":{"date-parts":[[2022,9,24]]},"reference":[{"key":"bibr1-01655515221112844","first-page":"3096","volume-title":"Proceedings of the 2020 IEEE international conference on big data","author":"Uchida H"},{"key":"bibr2-01655515221112844","first-page":"1","volume-title":"Proceedings of the 18th international conference on evaluation and assessment in software engineering","author":"Wohlin C."},{"key":"bibr3-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3375624"},{"key":"bibr4-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"bibr5-01655515221112844","doi-asserted-by":"publisher","DOI":"10.3233\/SW-190374"},{"key":"bibr6-01655515221112844","doi-asserted-by":"publisher","DOI":"10.5772\/intechopen.92433"},{"key":"bibr7-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-37439-6"},{"key":"bibr8-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-12375-8"},{"key":"bibr9-01655515221112844","unstructured":"Bresso E, Monnin P, Bousquet C et al. Investigating ADR mechanisms with knowledge graph mining and explainable AI, 2020, https:\/\/arxiv.org\/abs\/2012.09077"},{"key":"bibr10-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-62466-8_9"},{"key":"bibr11-01655515221112844","first-page":"1","volume":"33","author":"Kitchenham B.","year":"2004","journal-title":"Procedures for performing systematic reviews"},{"key":"bibr12-01655515221112844","first-page":"1","volume":"2465","author":"Seeliger A","year":"2019","journal-title":"PROFILES\/SEMEX@ISWC"},{"key":"bibr13-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12228"},{"key":"bibr14-01655515221112844","unstructured":"Benchekroun O, Rahimi A, Zhang Q et al. The need for standardized explainability, 2020, https:\/\/arxiv.org\/abs\/2010.11273"},{"key":"bibr15-01655515221112844","unstructured":"Molnar C. Interpretable machine learning, 2020, https:\/\/originalstatic.aminer.cn\/misc\/pdf\/Molnar-interpretable-machine-learning_compressed.pdf"},{"key":"bibr16-01655515221112844","volume-title":"The discipline of machine learning","volume":"9","author":"Mitchell TM.","year":"2006"},{"key":"bibr17-01655515221112844","first-page":"0200030","volume":"2","author":"Cruz JA","year":"2006","journal-title":"Canc Inform"},{"key":"bibr18-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2019.08.030"},{"key":"bibr19-01655515221112844","doi-asserted-by":"publisher","DOI":"10.3389\/fgene.2020.625659"},{"key":"bibr20-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-021-01518-6"},{"key":"bibr21-01655515221112844","doi-asserted-by":"publisher","DOI":"10.3389\/fbioe.2020.00867"},{"key":"bibr22-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33016666"},{"key":"bibr23-01655515221112844","doi-asserted-by":"publisher","DOI":"10.3233\/JIFS-189332"},{"key":"bibr24-01655515221112844","unstructured":"Yu J, McCluskey K, Mukherjee S. Tax knowledge graph for a smarter and more personalized turbotax, 2020, https:\/\/arxiv.org\/ftp\/arxiv\/papers\/2009\/2009.06103.pdf"},{"key":"bibr25-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317701"},{"key":"bibr26-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2018.12.008"},{"key":"bibr27-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2017.12.054"},{"key":"bibr28-01655515221112844","unstructured":"Silva VS, Freitas A, Handschuh S. XTE: explainable text entailment, 2020, https:\/\/arxiv.org\/pdf\/2009.12431.pdf"},{"key":"bibr29-01655515221112844","doi-asserted-by":"crossref","unstructured":"Nikolov A, d\u2019Aquin M. Uncovering semantic bias in neural network models using a knowledge graph. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp. 1175\u20131184, https:\/\/hal.archives-ouvertes.fr\/hal-03659110\/document","DOI":"10.1145\/3340531.3412009"},{"key":"bibr30-01655515221112844","first-page":"16495","volume":"1384","author":"Xu Y","year":"2020","journal-title":"Adv Neur Inform Process Syst"},{"key":"bibr31-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2020.107360"},{"key":"bibr32-01655515221112844","unstructured":"Geng Y, Chen J, Ye Z et al. Explainable zero-shot learning via attentive graph convolutional network and knowledge graphs, http:\/\/www.semantic-web-journal.net\/system\/files\/swj2318.pdf"},{"key":"bibr33-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1117\/12.2558083"},{"key":"bibr34-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372855"},{"issue":"1","key":"bibr35-01655515221112844","first-page":"39","volume":"56","author":"Fuji M","year":"2020","journal-title":"Fujit Sci Tech J"},{"key":"bibr36-01655515221112844","doi-asserted-by":"publisher","DOI":"10.2196\/17653"},{"key":"bibr37-01655515221112844","first-page":"113","volume-title":"Proceedings of the international conference on big data analytics","author":"Talukder AK"},{"key":"bibr38-01655515221112844","doi-asserted-by":"publisher","DOI":"10.2196\/25304"},{"key":"bibr39-01655515221112844","first-page":"2517","volume-title":"Proceedings of the 28th ACM international conference on information and knowledge management","author":"Luo X"},{"key":"bibr40-01655515221112844","first-page":"33","volume-title":"Proceedings of the 3rd international conference on machine learning and soft computing","author":"Zhao F"},{"key":"bibr41-01655515221112844","first-page":"7023","volume-title":"Proceedings of the AAAI conference on artificial intelligence","volume":"33","author":"Silva VS"},{"key":"bibr42-01655515221112844","first-page":"492","volume-title":"Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining","author":"Cui L"},{"key":"bibr43-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106389"},{"key":"bibr44-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1007\/s10844-018-0511-x"},{"key":"bibr45-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2020.3031769"},{"key":"bibr46-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2020.2979620"},{"key":"bibr47-01655515221112844","doi-asserted-by":"publisher","DOI":"10.3390\/info11020122"},{"key":"bibr48-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-61527-7_25"},{"key":"bibr49-01655515221112844","unstructured":"Ma L, Ma X, Gao J et al. CovidCare: transferring knowledge from existing EMR to emerging epidemic for interpretable prognosis, 2020, https:\/\/arxiv.org\/abs\/2007.08848"},{"key":"bibr50-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.addma.2020.101620"},{"key":"bibr51-01655515221112844","unstructured":"Ro\u017eanec JM, Mladeni\u0107 D. Semantic XAI for contextualized demand forecasting explanations, 2021, https:\/\/arxiv.org\/pdf\/2104.00452.pdf"},{"key":"bibr52-01655515221112844","first-page":"649","volume-title":"Proceedings of the 28th ACM international conference on information and knowledge management","author":"Xie X"},{"key":"bibr53-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30793-6_3"},{"key":"bibr54-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2020.103968"},{"key":"bibr55-01655515221112844","unstructured":"Chatterjee J, Dethlefs N. XAI4Wind: a multimodal knowledge graph database for explainable decision support in operations & maintenance of wind turbines, 2020, https:\/\/arxiv.org\/abs\/2012.10489"},{"key":"bibr56-01655515221112844","doi-asserted-by":"publisher","DOI":"10.5121\/csit.2020.101124"},{"key":"bibr57-01655515221112844","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103471"},{"issue":"2","key":"bibr58-01655515221112844","first-page":"58","volume":"55","author":"Fuji M","year":"2019","journal-title":"Fujit Sci Tech J"}],"container-title":["Journal of Information Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01655515221112844","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/01655515221112844","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01655515221112844","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T11:33:09Z","timestamp":1740915189000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/01655515221112844"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,24]]},"references-count":58,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.1177\/01655515221112844"],"URL":"https:\/\/doi.org\/10.1177\/01655515221112844","relation":{},"ISSN":["0165-5515","1741-6485"],"issn-type":[{"value":"0165-5515","type":"print"},{"value":"1741-6485","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,24]]}}}