{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,10]],"date-time":"2026-05-10T10:11:52Z","timestamp":1778407912033,"version":"3.51.4"},"reference-count":52,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T00:00:00Z","timestamp":1728000000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2024,10,4]]},"abstract":"<jats:p>In the past decade, trustworthy Artificial Intelligence (AI) has emerged as a focus for the AI community to ensure better adoption of AI models, and explainable AI is a cornerstone in this area. Over the years, the focus has shifted from building transparent AI methods to making recommendations on how to make black-box or opaque machine learning models and their results more understandable by experts and non-expert users. In our previous work, to address the goal of supporting user-centered explanations that make model recommendations more explainable, we developed an Explanation Ontology (EO). The EO is a general-purpose representation that was designed to help system designers connect explanations to their underlying data and knowledge. This paper addresses the apparent need for improved interoperability to support a wider range of use cases. We expand the EO, mainly in the system attributes contributing to explanations, by introducing new classes and properties to support a broader range of state-of-the-art explainer models. We present the expanded ontology model, highlighting the classes and properties that are important to model a larger set of fifteen literature-backed explanation types that are supported within the expanded EO. We build on these explanation type descriptions to show how to utilize the EO model to represent explanations in five use cases spanning the domains of finance, food, and healthcare. We include competency questions that evaluate the EO\u2019s capabilities to provide guidance for system designers on how to apply our ontology to their own use cases. This guidance includes allowing system designers to query the EO directly and providing them exemplar queries to explore content in the EO represented use cases. We have released this significantly expanded version of the Explanation Ontology at https:\/\/purl.org\/heals\/eo and updated our resource website, https:\/\/tetherless-world.github.io\/explanation-ontology, with supporting documentation. Overall, through the EO model, we aim to help system designers be better informed about explanations and support these explanations that can be composed, given their systems\u2019 outputs from various AI models, including a mix of machine learning, logical and explainer models, and different types of data and knowledge available to their systems.<\/jats:p>","DOI":"10.3233\/sw-233282","type":"journal-article","created":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T12:32:16Z","timestamp":1684499536000},"page":"959-989","source":"Crossref","is-referenced-by-count":15,"title":["Explanation Ontology: A general-purpose, semantic representation for supporting user-centered explanations"],"prefix":"10.1177","volume":"15","author":[{"given":"Shruthi","family":"Chari","sequence":"first","affiliation":[{"name":"Computer Science, Rensselaer Polytechnic Institute, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Oshani","family":"Seneviratne","sequence":"additional","affiliation":[{"name":"Computer Science, Rensselaer Polytechnic Institute, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohamed","family":"Ghalwash","sequence":"additional","affiliation":[{"name":"Center for Computational Health, IBM Research, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sola","family":"Shirai","sequence":"additional","affiliation":[{"name":"Computer Science, Rensselaer Polytechnic Institute, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel M.","family":"Gruen","sequence":"additional","affiliation":[{"name":"Computer Science, Rensselaer Polytechnic Institute, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pablo","family":"Meyer","sequence":"additional","affiliation":[{"name":"Center for Computational Health, IBM Research, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Prithwish","family":"Chakraborty","sequence":"additional","affiliation":[{"name":"Center for Computational Health, IBM Research, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Deborah L.","family":"McGuinness","sequence":"additional","affiliation":[{"name":"Computer Science, Rensselaer Polytechnic Institute, NY, US"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/SW-233282_ref1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jbi.2018.02.010","article-title":"Assessing the practice of biomedical ontology evaluation: Gaps and opportunities","volume":"80","author":"Amith","year":"2018","journal-title":"Journal of biomedical informatics"},{"key":"10.3233\/SW-233282_ref2","doi-asserted-by":"crossref","unstructured":"V.\u00a0Arya, R.K.\u00a0Bellamy, P.-Y.\u00a0Chen, A.\u00a0Dhurandhar, M.\u00a0Hind, S.C.\u00a0Hoffman, S.\u00a0Houde, Q.V.\u00a0Liao, R.\u00a0Luss, A.\u00a0Mojsilovi\u0107 et al., AI explainability 360: Impact and design, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol.\u00a036, 2022, pp.\u00a012651\u201312657.","DOI":"10.1609\/aaai.v36i11.21540"},{"key":"10.3233\/SW-233282_ref4","doi-asserted-by":"crossref","unstructured":"D.\u00a0Bau, B.\u00a0Zhou, A.\u00a0Khosla, A.\u00a0Oliva and A.\u00a0Torralba, Network dissection: Quantifying interpretability of deep visual representations, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 2017, pp.\u00a06541\u20136549.","DOI":"10.1109\/CVPR.2017.354"},{"key":"10.3233\/SW-233282_ref6","unstructured":"J.\u00a0Brank, M.\u00a0Grobelnik and D.\u00a0Mladenic, A survey of ontology evaluation techniques, in: Proceedings of the Conference on Data Mining and Data Warehouses (SiKDD 2005), Citeseer, Ljubljana Slovenia, 2005, pp.\u00a0166\u2013170."},{"key":"10.3233\/SW-233282_ref7","unstructured":"Cancer Biomedical Informatics Grid, Unified Medical Language System, National Cancer Institute Thesarus (NCIT)."},{"key":"10.3233\/SW-233282_ref8","unstructured":"Center for Disease Control and Prevention (CDC), National Health and Nutrition Examination Survey."},{"issue":"24","key":"10.3233\/SW-233282_ref9","doi-asserted-by":"publisher","first-page":"2405","DOI":"10.1001\/jama.2019.5284","article-title":"The proliferation of reports on clinical scoring systems: Issues about uptake and clinical utility","volume":"321","author":"Challener","year":"2019","journal-title":"Jama"},{"key":"10.3233\/SW-233282_ref11","unstructured":"S.\u00a0Chari, D.M.\u00a0Gruen, O.\u00a0Seneviratne and D.L.\u00a0McGuinness, Directions for explainable knowledge-enabled systems, in: Knowledge Graphs for eXplainable AI\u00a0\u2013 Foundations, Applications and Challenges, Studies on the Semantic Web, I.\u00a0Tiddi, F.\u00a0Lecue and P.\u00a0Hitzler, eds, IOS Press, 2020, to appear."},{"key":"10.3233\/SW-233282_ref12","unstructured":"S.\u00a0Chari, D.M.\u00a0Gruen, O.\u00a0Seneviratne and D.L.\u00a0McGuinness, Foundations of explainable knowledge-enabled systems, in: Knowledge Graphs for eXplainable AI\u00a0\u2013 Foundations, Applications and Challenges, Studies on the Semantic Web, I.\u00a0Tiddi, F.\u00a0Lecue and P.\u00a0Hitzler, eds, IOS Press, 2020, to appear."},{"key":"10.3233\/SW-233282_ref13","doi-asserted-by":"crossref","unstructured":"S.\u00a0Chari, O.\u00a0Seneviratne, D.M.\u00a0Gruen, M.A.\u00a0Foreman, A.K.\u00a0Das and D.L.\u00a0McGuinness, Explanation ontology: A model of explanations for user-centered ai, in: International Semantic Web Conference, Springer, 2020, pp.\u00a0228\u2013243.","DOI":"10.1007\/978-3-030-62466-8_15"},{"key":"10.3233\/SW-233282_ref14","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441816"},{"issue":"1","key":"10.3233\/SW-233282_ref15","doi-asserted-by":"publisher","first-page":"23","DOI":"10.3233\/AO-2011-0087","article-title":"MIREOT: The minimum information to reference an external ontology term","volume":"6","author":"Courtot","year":"2011","journal-title":"Applied Ontology"},{"issue":"1","key":"10.3233\/SW-233282_ref17","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/S0950-7051(98)00053-7","article-title":"CyberDesk: A framework for providing self-integrating context-aware services","volume":"11","author":"Dey","year":"1998","journal-title":"Knowledge-based systems"},{"key":"10.3233\/SW-233282_ref19","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2022.100493"},{"key":"10.3233\/SW-233282_ref22","doi-asserted-by":"publisher","DOI":"10.1186\/2041-1480-5-14"},{"key":"10.3233\/SW-233282_ref23","unstructured":"FICO Community, Explainable Machine Learning Challenge, MITr."},{"issue":"1","key":"10.3233\/SW-233282_ref24","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1109\/MIC.2020.3031769","article-title":"Semantics of the Black-Box: Can knowledge graphs help make deep learning systems more interpretable and explainable?","volume":"25","author":"Gaur","year":"2021","journal-title":"IEEE Internet Computing"},{"issue":"11","key":"10.3233\/SW-233282_ref25","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","year":"2021","journal-title":"The Lancet Digital Health"},{"issue":"11","key":"10.3233\/SW-233282_ref26","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","year":"2021","journal-title":"The Lancet Digital Health"},{"key":"10.3233\/SW-233282_ref28","unstructured":"D.M.\u00a0Gruen, S.\u00a0Chari, M.A.\u00a0Foreman, O.\u00a0Seneviratne, R.\u00a0Richesson, A.K.\u00a0Das and D.L.\u00a0McGuinness, Designing for AI explainability in clinical context, in: Trustworthy AI for Healthcare Workshop at AAAI 2021, 2021."},{"key":"10.3233\/SW-233282_ref29","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2019.00036"},{"key":"10.3233\/SW-233282_ref30","doi-asserted-by":"crossref","unstructured":"S.\u00a0Haussmann, O.\u00a0Seneviratne, Y.\u00a0Chen, Y.\u00a0Ne\u2019eman, J.\u00a0Codella, C.-H.\u00a0Chen, D.L.\u00a0McGuinness and M.J.\u00a0Zaki, FoodKG: A semantics-driven knowledge graph for food recommendation, in: International Semantic Web Conference, Springer, 2019, pp.\u00a0146\u2013162.","DOI":"10.1007\/978-3-030-30796-7_10"},{"key":"10.3233\/SW-233282_ref31","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314273"},{"key":"10.3233\/SW-233282_ref32","unstructured":"IBM Research Trusted AI, AI Explainability 360 Open Source Toolkit."},{"issue":"1","key":"10.3233\/SW-233282_ref33","doi-asserted-by":"publisher","first-page":"i","DOI":"10.1007\/978-3-031-79486-5","article-title":"Ontology engineering","volume":"9","author":"Kendall","year":"2019","journal-title":"Synthesis Lectures on The Semantic Web: Theory and Technology"},{"issue":"1","key":"10.3233\/SW-233282_ref35","doi-asserted-by":"publisher","first-page":"41","DOI":"10.3233\/SW-190374","article-title":"On the role of knowledge graphs in explainable AI","volume":"11","author":"Lecue","year":"2020","journal-title":"Semantic Web"},{"key":"10.3233\/SW-233282_ref36","doi-asserted-by":"crossref","unstructured":"Q.V.\u00a0Liao, D.\u00a0Gruen and S.\u00a0Miller, Questioning the AI: Informing design practices for explainable AI user experiences, in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp.\u00a01\u201315.","DOI":"10.1145\/3313831.3376590"},{"key":"10.3233\/SW-233282_ref37","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1519023"},{"issue":"4","key":"10.3233\/SW-233282_ref39","doi-asserted-by":"publisher","first-page":"397","DOI":"10.1016\/j.websem.2004.06.002","article-title":"Explaining answers from the semantic web: The inference web approach","volume":"1","author":"McGuinness","year":"2004","journal-title":"Web Semantics: Sci., Services and Agents on the World Wide Web"},{"key":"10.3233\/SW-233282_ref40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in artificial intelligence: Insights from the social sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artificial Intelligence"},{"key":"10.3233\/SW-233282_ref41","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287574"},{"key":"10.3233\/SW-233282_ref42","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1007\/s11257-017-9195-0","article-title":"A systematic review and taxonomy of explanations in decision support and recommender systems","volume":"27","author":"Nunes","year":"2017","journal-title":"User Modeling and User-Adapted Interaction"},{"key":"10.3233\/SW-233282_ref43","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-48438-9_8"},{"key":"10.3233\/SW-233282_ref44","unstructured":"Ontograf\u00a0\u2013 Protege Wiki, Accessed: 2022-04-04."},{"key":"10.3233\/SW-233282_ref45","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW53142.2021.00010"},{"issue":"3","key":"10.3233\/SW-233282_ref46","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1093\/jamiaopen\/ooaa033","article-title":"Evaluating artificial intelligence in medicine: Phases of clinical research","volume":"3","author":"Park","year":"2020","journal-title":"JAMIA open"},{"key":"10.3233\/SW-233282_ref47","doi-asserted-by":"publisher","DOI":"10.1145\/3450439.3451869"},{"key":"10.3233\/SW-233282_ref48","doi-asserted-by":"crossref","unstructured":"M.T.\u00a0Ribeiro, S.\u00a0Singh and C.\u00a0Guestrin, Why should I trust you?: Explaining the predictions of any classifier, in: Proc. of the 22nd ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, ACM, 2016, pp.\u00a01135\u20131144.","DOI":"10.1145\/2939672.2939778"},{"key":"10.3233\/SW-233282_ref49","unstructured":"S.C. for Biomedical Research (BMIR), Protege."},{"key":"10.3233\/SW-233282_ref50","unstructured":"S.C. for Biomedical Research (BMIR), Ontology Metrics."},{"key":"10.3233\/SW-233282_ref51","doi-asserted-by":"crossref","unstructured":"A.A.\u00a0Salatino, T.\u00a0Thanapalasingam, A.\u00a0Mannocci, F.\u00a0Osborne and E.\u00a0Motta, The computer science ontology: A large-scale taxonomy of research areas, in: International Semantic Web Conference, Springer, 2018, pp.\u00a0187\u2013205.","DOI":"10.1007\/978-3-030-00668-6_12"},{"issue":"11","key":"10.3233\/SW-233282_ref52","doi-asserted-by":"publisher","first-page":"1251","DOI":"10.1038\/nbt1346","article-title":"The OBO foundry: Coordinated evolution of ontologies to support biomedical data integration","volume":"25","author":"Smith","year":"2007","journal-title":"Nature biotechnology"},{"issue":"3\u20134","key":"10.3233\/SW-233282_ref53","doi-asserted-by":"publisher","first-page":"139","DOI":"10.3233\/AO-2010-0079","article-title":"Ontological realism: A methodology for coordinated evolution of scientific ontologies","volume":"5","author":"Smith","year":"2010","journal-title":"Applied ontology"},{"key":"10.3233\/SW-233282_ref54","doi-asserted-by":"publisher","DOI":"10.1007\/978-90-481-8847-5_5"},{"key":"10.3233\/SW-233282_ref56","unstructured":"The Apache Software Foundation, Vol.\u00a02.0 License."},{"key":"10.3233\/SW-233282_ref57","doi-asserted-by":"crossref","unstructured":"I.\u00a0Tiddi, M.\u00a0d\u2019Aquin and E.\u00a0Motta, Dedalo: Looking for clusters explanations in a labyrinth of linked data, in: European Semantic Web Conf., Springer, 2014, pp.\u00a0333\u2013348.","DOI":"10.1007\/978-3-319-07443-6_23"},{"key":"10.3233\/SW-233282_ref58","doi-asserted-by":"crossref","unstructured":"I.\u00a0Tiddi, M.\u00a0d\u2019Aquin and E.\u00a0Motta, An ontology design pattern to define explanations, in: Proceedings of the 8th Int. Conf. on Knowledge Capture, 2015, pp.\u00a01\u20138.","DOI":"10.1145\/2815833.2815844"},{"key":"10.3233\/SW-233282_ref59","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103627"},{"key":"10.3233\/SW-233282_ref60","unstructured":"S.\u00a0Tonekaboni, S.\u00a0Joshi, M.D.\u00a0McCradden and A.\u00a0Goldenberg, What clinicians want: Contextualizing explainable machine learning for clinical end use, in: Machine Learning for Healthcare Conference, PMLR, 2019, pp.\u00a0359\u2013380."},{"key":"10.3233\/SW-233282_ref62","doi-asserted-by":"crossref","unstructured":"S.\u00a0Wachter, B.\u00a0Mittelstadt and C.\u00a0Russell, Counterfactual explanations without opening the black box: Automated decisions and the GPDR, Harv. JL & Tech. 31 (2017), 841.","DOI":"10.2139\/ssrn.3063289"},{"key":"10.3233\/SW-233282_ref63","doi-asserted-by":"crossref","unstructured":"D.\u00a0Wang, Q.\u00a0Yang, A.\u00a0Abdul and B.Y.\u00a0Lim, Designing theory-driven user-centric explainable AI, in: Proceedings of the 2019 CHI Conf. on Human Factors in Computing Systems, 2019, pp.\u00a01\u201315.","DOI":"10.1145\/3290605.3300831"},{"key":"10.3233\/SW-233282_ref64","doi-asserted-by":"publisher","DOI":"10.3390\/electronics10050593"}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-233282","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:26:54Z","timestamp":1777613214000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-233282"}},"subtitle":[],"editor":[{"given":"Roberto","family":"Confalonieri","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Oliver","family":"Kutz","sequence":"additional","affiliation":[{"name":"Free University of Bozen-Bolzano, Italy"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Diego","family":"Calvanese","sequence":"additional","affiliation":[{"name":"Ume\u00e5 University, Sweden and Free University of Bozen-Bolzano, Italy"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Jose M.","family":"Alonso","sequence":"additional","affiliation":[{"name":"University of Santiago de Compostela, CiTIUS, Spain"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Shang-Ming","family":"Zhou","sequence":"additional","affiliation":[{"name":"University of Plymouth, UK"}],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,10,4]]},"references-count":52,"journal-issue":{"issue":"4"},"URL":"https:\/\/doi.org\/10.3233\/sw-233282","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"value":"2210-4968","type":"electronic"},{"value":"1570-0844","type":"print"}],"subject":[],"published":{"date-parts":[[2024,10,4]]}}}