{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T14:09:40Z","timestamp":1783433380529,"version":"3.54.6"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2021,2,7]],"date-time":"2021-02-07T00:00:00Z","timestamp":1612656000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,7]],"date-time":"2021-02-07T00:00:00Z","timestamp":1612656000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Explanation mechanisms for intelligent systems are typically designed to respond to specific user needs, yet in practice these systems tend to have a wide variety of users. This can present a challenge to organisations looking to satisfy the explanation needs of different groups using an individual system. In this paper we present an explainability framework formed of a catalogue of explanation methods, and designed to integrate with a range of projects within a telecommunications organisation. Explainability methods are split into low-level explanations and high-level explanations for increasing levels of contextual support in their explanations. We motivate this framework using the specific case-study of explaining the conclusions of field network engineering experts to non-technical planning staff and evaluate our results using feedback from two distinct user groups; domain-expert telecommunication engineers and non-expert desk agent staff. We also present and investigate two metrics designed to model the quality of explanations - Meet-In-The-Middle (MITM) and Trust-Your-Neighbours (TYN). Our analysis of these metrics offers new insights into the use of similarity knowledge for the evaluation of explanations.<\/jats:p>","DOI":"10.1007\/s13218-020-00702-6","type":"journal-article","created":{"date-parts":[[2021,2,9]],"date-time":"2021-02-09T04:23:49Z","timestamp":1612844629000},"page":"397-411","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Evaluating Explainability Methods Intended for Multiple Stakeholders"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0941-3111","authenticated-orcid":false,"given":"Kyle","family":"Martin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anne","family":"Liret","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nirmalie","family":"Wiratunga","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gilbert","family":"Owusu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mathias","family":"Kern","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,2,7]]},"reference":[{"issue":"8","key":"702_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0181142","volume":"12","author":"L Arras","year":"2017","unstructured":"Arras L, Horn F, Montavon G, M\u00fcller KR, Samek W (2017) What is relevant in a text document ? An interpretable machine learning approach. PLOS One 12(8):1\u201323. https:\/\/doi.org\/10.1371\/journal.pone.0181142","journal-title":"PLOS One"},{"key":"702_CR2","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garc\u00eda S, Gil-L\u00f3pez S, Molina D, Benjamins R et al (2020) Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. Inform Fusion 58:82\u2013115","journal-title":"Inform Fusion"},{"key":"702_CR3","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1007\/3-540-44527-7_3","volume-title":"Advances in case-based reasoning","author":"W Cheetham","year":"2000","unstructured":"Cheetham W (2000) Case-based reasoning with confidence. In: Blanzieri E, Portinale L (eds) Advances in case-based reasoning. Springer, Berlin, pp 15\u201325"},{"key":"702_CR4","doi-asserted-by":"crossref","unstructured":"Collins E, Augenstein, I, Riedel S (2017) A supervised approach to extractive summarisation of scientific papers. CoNLL 2017-21st Conference on Computational Natural Language Learning, Proceedings. Association for Computational Linguistics (ACL), pp 195\u2013205","DOI":"10.18653\/v1\/K17-1021"},{"key":"702_CR5","unstructured":"Regulation (EU) 2016\/679 of the European Parliament and of the Council. (2016)"},{"key":"702_CR6","doi-asserted-by":"crossref","unstructured":"Fong RC, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. In: Proceedings of the IEEE International Conference on Computer Vision, pp 3429\u20133437","DOI":"10.1109\/ICCV.2017.371"},{"key":"702_CR7","doi-asserted-by":"crossref","unstructured":"Gilpin LH, Bau D, Yuan BZ, Bajwa A, Specter M, Kagal L (2018) Explaining explanations: an overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), IEEE, pp 80\u201389","DOI":"10.1109\/DSAA.2018.00018"},{"key":"702_CR8","volume-title":"Explainable artificial intelligence (xai)","author":"D Gunning","year":"2017","unstructured":"Gunning D (2017) Explainable artificial intelligence (xai). Defense Advanced Research Projects Agency (DARPA), Arlington"},{"issue":"4","key":"702_CR9","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1007\/s10506-007-9039-z","volume":"14","author":"B Hachey","year":"2006","unstructured":"Hachey B, Grover C (2006) Extractive summarisation of legal texts. Artif Intell Law 14(4):305\u2013345","journal-title":"Artif Intell Law"},{"key":"702_CR10","unstructured":"Hoffman RR, Mueller ST, Klein G, Litman J (2018) Metrics for explainable ai: challenges and prospects. arXiv preprint arXiv:1812.04608"},{"key":"702_CR11","doi-asserted-by":"crossref","unstructured":"Hou YL, Peng J, Hao X, Shen Y, Qian M (2017) Occlusion localization based on convolutional neural networks. In: 2017 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), IEEE, pp 1\u20135","DOI":"10.1109\/ICSPCC.2017.8242508"},{"key":"702_CR12","unstructured":"Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980"},{"key":"702_CR13","doi-asserted-by":"crossref","unstructured":"Kulesza T, Stumpf S, Burnett M, Wong WK, Riche Y, Moore T, Oberst I, Shinsel A, McIntosh K (2010)Explanatory debugging: supporting end-user debugging of machine-learned programs. In: 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, IEEE, pp 41\u201348","DOI":"10.1109\/VLHCC.2010.15"},{"key":"702_CR14","unstructured":"Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML\u201914. JMLR.org, pp II1188\u2013II1196"},{"key":"702_CR15","volume-title":"Case-Based Reasoning: experiences. Lessons and future directions","author":"DB Leake","year":"1996","unstructured":"Leake DB (1996) Case-Based Reasoning: experiences. Lessons and future directions. MIT Press, Cambridge"},{"key":"702_CR16","doi-asserted-by":"crossref","unstructured":"Lind M, Johansson J, Cooper M (2009) Many-to-many relational parallel coordinates displays. In: 2009 13th International Conference on Information Visualisation, pp 25\u201331","DOI":"10.1109\/IV.2009.43"},{"key":"702_CR17","doi-asserted-by":"crossref","unstructured":"Lipton, Zachary C (2016) The mythos of model interpretability.Queue, pp 31\u201357","DOI":"10.1145\/3236386.3241340"},{"issue":"2","key":"702_CR18","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1147\/rd.22.0159","volume":"2","author":"HP Luhn","year":"1958","unstructured":"Luhn HP (1958) The automatic creation of literature abstracts. IBM J Res Dev 2(2):159\u2013165","journal-title":"IBM J Res Dev"},{"key":"702_CR19","unstructured":"Massie S, Craw S, Wiratunga W (2004) Visualisation of case-base reasoning for explanation. In: Proceedings of the ECCBR 2004 Workshops, pp 135\u2013144"},{"key":"702_CR20","unstructured":"Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781"},{"key":"702_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2018","unstructured":"Miller T (2018) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338","journal-title":"Artif Intell"},{"key":"702_CR22","unstructured":"Mohseni S, Zarei N, Eric D Ragan (2018) A survey of evaluation methods and measures for interpretable machine learning. arXiv preprint arXiv:1811.11839"},{"key":"702_CR23","unstructured":"Mueller ST, Hoffman RR, Clancey W, Emrey A, Klein G (2019) Explanation in human-ai systems: a literature meta-review, synopsis of key ideas and publications, and bibliography for explainable ai."},{"key":"702_CR24","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1007\/978-3-319-61030-6_16","volume-title":"Case-based reasoning research and development","author":"K Muhammad","year":"2017","unstructured":"Muhammad K, Lawlor A, Smyth B (2017) On the pros and cons of explanation-based ranking. In: Aha DW, Lieber J (eds) Case-based reasoning research and development. Springer International Publishing, Cham, pp 227\u2013241"},{"issue":"11","key":"702_CR25","doi-asserted-by":"publisher","first-page":"1905","DOI":"10.1080\/00140139408964957","volume":"37","author":"BM Muir","year":"1994","unstructured":"Muir BM (1994) Trust in automation: part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics 37(11):1905\u20131922","journal-title":"Ergonomics"},{"issue":"3","key":"702_CR26","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1023\/A:1026446214010","volume":"3","author":"I Nordin","year":"2000","unstructured":"Nordin I (2000) Expert and non-expert knowledge in medical practice. Med Health Care Philos 3(3):295\u2013302. https:\/\/doi.org\/10.1023\/A:1026446214010","journal-title":"Med Health Care Philos"},{"key":"702_CR27","unstructured":"Ramos J (2003) Using tf-idf to determine word relevance in document queries. In: Proceedings of the first instructional conference on machine learning, pp 133 \u2013 142"},{"key":"702_CR28","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1007\/978-3-319-98131-4_2","volume-title":"Explanation methods in deep learning: users values, concerns and challenges","author":"G Ras","year":"2018","unstructured":"Ras G, van Gerven M, Haselager P (2018) Explanation methods in deep learning: users values, concerns and challenges. Springer International Publishing, Cham, pp 19\u201336. https:\/\/doi.org\/10.1007\/978-3-319-98131-4_2"},{"key":"702_CR29","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u201d why should i trust you?\u201d explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"issue":"6","key":"702_CR30","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1007\/s10458-019-09408-y","volume":"33","author":"A Rosenfeld","year":"2019","unstructured":"Rosenfeld A, Richardson A (2019) Explainability in human-agent systems. Auton Agents Multi-Agent Syst 33(6):673\u2013705","journal-title":"Auton Agents Multi-Agent Syst"},{"key":"702_CR31","doi-asserted-by":"crossref","unstructured":"Ross AS, Hughes MC, Doshi-Velez F (2017) Right for the right reasons: training differentiable models by constraining their explanations. Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp 2662\u20132670","DOI":"10.24963\/ijcai.2017\/371"},{"key":"702_CR32","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1007\/978-3-540-28631-8_29","volume-title":"Advances in case-based reasoning","author":"TR Roth-Berghofer","year":"2004","unstructured":"Roth-Berghofer TR (2004) Explanations and case-based reasoning: foundational issues. In: Funk P, Gonz\u00e1lez PAC (eds) Advances in case-based reasoning. Springer, Berlin, pp 389\u2013403"},{"key":"702_CR33","unstructured":"S\u00f8rmo F, Cassens J (2004) Explanation goals in case-based reasoning. In: Proceedings of the ECCBR, pp 165\u2013174"},{"issue":"2","key":"702_CR34","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1007\/s10462-005-4607-7","volume":"24","author":"F S\u00f8rmo","year":"2005","unstructured":"S\u00f8rmo F, Cassens J, Aamodt A (2005) Explanation in case-based reasoning: perspectives and goals. Artif Intell Rev 24(2):109\u2013143","journal-title":"Artif Intell Rev"},{"key":"702_CR35","doi-asserted-by":"crossref","unstructured":"Zhang Y, Sreedharan S, Kulkarni A, Chakraborti T, Zhuo HH, Kambhampati S (2017) Plan explicability and predictability for robot task planning. In: 2017 IEEE international conference on robotics and automation (ICRA), IEEE, pp 1313\u20131320","DOI":"10.1109\/ICRA.2017.7989155"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00702-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13218-020-00702-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00702-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T08:25:51Z","timestamp":1636964751000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13218-020-00702-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,7]]},"references-count":35,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["702"],"URL":"https:\/\/doi.org\/10.1007\/s13218-020-00702-6","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,7]]},"assertion":[{"value":"3 April 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 December 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 February 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}