{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T15:40:47Z","timestamp":1773330047438,"version":"3.50.1"},"reference-count":149,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,11,14]],"date-time":"2022-11-14T00:00:00Z","timestamp":1668384000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,14]],"date-time":"2022-11-14T00:00:00Z","timestamp":1668384000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004115","name":"Gottfried Wilhelm Leibniz Universit\u00e4t Hannover","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004115","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Requirements Eng"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The growing complexity of software systems and the influence of software-supported decisions in our society sparked the need for software that is transparent, accountable, and trustworthy. Explainability has been identified as a means to achieve these qualities. It is recognized as an emerging non-functional requirement (NFR) that has a significant impact on system quality. Accordingly, software engineers need means to assist them in incorporating this NFR into systems. This requires an early analysis of the benefits and possible design issues that arise from interrelationships between different quality aspects. However, explainability is currently under-researched in the domain of requirements engineering, and there is a lack of artifacts that support the requirements engineering process and system design. In this work, we remedy this deficit by proposing four artifacts: a definition of explainability, a conceptual model, a knowledge catalogue, and a reference model for explainable systems. These artifacts should support software and requirements engineers in understanding the definition of explainability and how it interacts with other quality aspects. Besides that, they may be considered a starting point to provide practical value in the refinement of explainability from high-level requirements to concrete design choices, as well as on the identification of methods and metrics for the evaluation of the implemented requirements.<\/jats:p>","DOI":"10.1007\/s00766-022-00393-5","type":"journal-article","created":{"date-parts":[[2022,11,14]],"date-time":"2022-11-14T20:16:48Z","timestamp":1668457008000},"page":"457-487","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Explainable software systems: from requirements analysis to system evaluation"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6093-8875","authenticated-orcid":false,"given":"Larissa","family":"Chazette","sequence":"first","affiliation":[]},{"given":"Wasja","family":"Brunotte","sequence":"additional","affiliation":[]},{"given":"Timo","family":"Speith","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,11,14]]},"reference":[{"key":"393_CR1","unstructured":"Mike walsh: welcome to the algorithmic age. Accessed 27 Jan 2022 (2018). https:\/\/www.mike-walsh.com\/news\/welcome-to-the-algorithmic-age"},{"key":"393_CR2","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1007\/978-1-4842-3799-1_6","volume-title":"Machine learning and AI for healthcare: big data for improved health outcomes","author":"A Panesar","year":"2019","unstructured":"Panesar A (2019) Ethics of intelligence. Machine learning and AI for healthcare: big data for improved health outcomes. Apress, New York, NY, pp 207\u2013254. https:\/\/doi.org\/10.1007\/978-1-4842-3799-1_6"},{"issue":"4","key":"393_CR3","doi-asserted-by":"publisher","first-page":"611","DOI":"10.1007\/s13347-017-0279-x","volume":"31","author":"B Lepri","year":"2018","unstructured":"Lepri B, Oliver N, Letouz\u00e9 E, Pentland A, Vinck P (2018) Fair, transparent, and accountable algorithmic decision-making processes. Philos Technol 31(4):611\u2013627. https:\/\/doi.org\/10.1007\/s13347-017-0279-x","journal-title":"Philos Technol"},{"key":"393_CR4","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi A, Berrada M (2018) Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6:52138\u201352160. https:\/\/doi.org\/10.1109\/ACCESS.2018.2870052","journal-title":"IEEE Access"},{"issue":"4","key":"393_CR5","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1007\/s00766-020-00333-1","volume":"25","author":"L Chazette","year":"2020","unstructured":"Chazette L, Schneider K (2020) Explainability as a non-functional requirement: challenges and recommendations. Requir Eng 25(4):493\u2013514. https:\/\/doi.org\/10.1007\/s00766-020-00333-1","journal-title":"Requir Eng"},{"key":"393_CR6","doi-asserted-by":"publisher","unstructured":"Langer M, Baum K, Hartmann K, Hessel S, Speith T, Wahl J (2021) Explainability auditing for intelligent systems: a rationale for multi-disciplinary perspectives. In: 2021 IEEE 29th international requirements engineering conference workshops (REW). pp 164\u2013168. https:\/\/doi.org\/10.1109\/REW53955.2021.00030","DOI":"10.1109\/REW53955.2021.00030"},{"key":"393_CR7","doi-asserted-by":"publisher","unstructured":"Glinz M (2007) On non-functional requirements. In: 15th IEEE international requirements engineering conference (RE). pp 21\u201326. https:\/\/doi.org\/10.1109\/RE.2007.45","DOI":"10.1109\/RE.2007.45"},{"key":"393_CR8","doi-asserted-by":"publisher","unstructured":"Chazette L, Karras O, Schneider K (2019) Do end-users want explanations? Analyzing the role of explainability as an emerging aspect of non-functional requirements. In: 2019 IEEE 27th international requirements engineering conference (RE). pp 223\u2013233. https:\/\/doi.org\/10.1109\/RE.2019.00032","DOI":"10.1109\/RE.2019.00032"},{"key":"393_CR9","doi-asserted-by":"publisher","unstructured":"K\u00f6hl MA, Baum K, Langer M, Oster D, Speith T, Bohlender D (2019) Explainability as a non-functional requirement. In: 27th IEEE international requirements engineering conference (RE). IEEE, New York, NY, pp 363\u2013368. https:\/\/doi.org\/10.1109\/RE.2019.00046","DOI":"10.1109\/RE.2019.00046"},{"key":"393_CR10","doi-asserted-by":"publisher","unstructured":"Bussone A, Stumpf S, O\u2019Sullivan D (2015) The role of explanations on trust and reliance in clinical decision support systems. In: 2015 international conference on healthcare informatics. IEEE, New York, NY, pp 160\u2013169. https:\/\/doi.org\/10.1109\/ICHI.2015.26","DOI":"10.1109\/ICHI.2015.26"},{"key":"393_CR11","doi-asserted-by":"publisher","unstructured":"Winkler JP, Vogelsang A (2017) \u201cWhat does my classifier learn?\u201d A visual approach to understanding natural language text classifiers. In: Frasincar F, Ittoo A, Nguyen LM, M\u00e9tais E (eds) Natural language and information systems. pp 468\u2013479. https:\/\/doi.org\/10.1007\/978-3-319-59569-6_55","DOI":"10.1007\/978-3-319-59569-6_55"},{"key":"393_CR12","doi-asserted-by":"publisher","unstructured":"Zhou J, Hu H, Li Z, Yu K, Chen F (2019) Physiological indicators for user trust in machine learning with influence enhanced fact-checking. In: International cross-domain conference for machine learning and knowledge extraction. Springer, Cham, pp 94\u2013113. https:\/\/doi.org\/10.1007\/978-3-030-29726-8_7","DOI":"10.1007\/978-3-030-29726-8_7"},{"key":"393_CR13","doi-asserted-by":"publisher","unstructured":"Hind M, Wei D, Campbell M, Codella NCF, Dhurandhar A, Mojsilovi\u0107 A, Natesan\u00a0RK, Varshney KR (2019) TED: teaching AI to explain its decisions. In: Proceedings of the 2019 AAAI\/ACM conference on AI, ethics, and society. ACM, New York, NY, pp 123\u2013129. https:\/\/doi.org\/10.1145\/3306618.3314273","DOI":"10.1145\/3306618.3314273"},{"issue":"6","key":"393_CR14","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1007\/s10458-019-09408-y","volume":"33","author":"A Rosenfeld","year":"2019","unstructured":"Rosenfeld A, Richardson A (2019) Explainability in human-agent systems. Auton Agent Multi-Agent Syst 33(6):673\u2013705. https:\/\/doi.org\/10.1007\/s10458-019-09408-y","journal-title":"Auton Agent Multi-Agent Syst"},{"key":"393_CR15","doi-asserted-by":"publisher","unstructured":"Brunotte W, Specht A, Chazette L, Schneider K (2022) Privacy explanations \u2013 a means to end-user trust. arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2210.09706","DOI":"10.48550\/ARXIV.2210.09706"},{"key":"393_CR16","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/978-3-319-90403-0_2","volume-title":"Human and machine learning: visible, explainable, trustworthy and transparent","author":"B Abdollahi","year":"2018","unstructured":"Abdollahi B, Nasraoui O (2018) Transparency in fair machine learning: the case of explainable recommender systems. Human and machine learning: visible, explainable, trustworthy and transparent. Springer, Cham, CH, pp 21\u201335. https:\/\/doi.org\/10.1007\/978-3-319-90403-0_2"},{"key":"393_CR17","doi-asserted-by":"publisher","unstructured":"Thiebes S, Lins S, Sunyaev A (2020) Trustworthy artificial intelligence. Electron Mark 1\u201318 . https:\/\/doi.org\/10.1007\/s12525-020-00441-4","DOI":"10.1007\/s12525-020-00441-4"},{"key":"393_CR18","doi-asserted-by":"publisher","unstructured":"Chazette L, Kl\u00f6s V, Herzog F, Schneider K (2022) Requirements on explanations: a quality framework for explainability. In: Proceedings of the 2022 IEEE 30th international requirements engineering conference (RE), pp 140\u2013152. https:\/\/doi.org\/10.1109\/RE54965.2022.00019","DOI":"10.1109\/RE54965.2022.00019"},{"issue":"3","key":"393_CR19","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1007\/s00450-014-0256-x","volume":"30","author":"M Glinz","year":"2015","unstructured":"Glinz M, Fricker SA (2015) On shared understanding in software engineering: an essay. Comput Sci Res Dev 30(3):363\u2013376. https:\/\/doi.org\/10.1007\/s00450-014-0256-x","journal-title":"Comput Sci Res Dev"},{"key":"393_CR20","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/978-3-642-23391-3_3","volume-title":"Evaluation of novel approaches to software engineering","author":"D Mairiza","year":"2011","unstructured":"Mairiza D, Zowghi D (2011) Constructing a catalogue of conflicts among non-functional requirements. Evaluation of novel approaches to software engineering. Springer, Berlin, Heidelberg, pp 31\u201344"},{"key":"393_CR21","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-5269-7","volume-title":"Non-functional requirements in software engineering","author":"L Chung","year":"2012","unstructured":"Chung L, Nixon BA, Yu E, Mylopoulos J (2012) Non-functional requirements in software engineering. Springer, Boston, MA. https:\/\/doi.org\/10.1007\/978-1-4615-5269-7"},{"key":"393_CR22","doi-asserted-by":"publisher","unstructured":"Gacit\u00faa R, Ma L, Nuseibeh B, Piwek P, Roeck AND, Rouncefield M, Sawyer P, Willis A, Yang H (2009) Making tacit requirements explicit. In: Second international workshop on managing requirements knowledge (MARK@RE). IEEE, New York, NY, pp 40\u201344. https:\/\/doi.org\/10.1109\/MARK.2009.7","DOI":"10.1109\/MARK.2009.7"},{"key":"393_CR23","doi-asserted-by":"publisher","unstructured":"Santos D, Resende A, Junior PA, Costa H (2016) Attributes and metrics of internal quality that impact the external quality of object-oriented software: a systematic literature review. In: 2016 XLII Latin American computing conference (CLEI). pp 1\u201312. https:\/\/doi.org\/10.1109\/CLEI.2016.7833322","DOI":"10.1109\/CLEI.2016.7833322"},{"key":"393_CR24","unstructured":"Paech B, Kerkow D (2004) Non-functional requirements engineering-quality is essential. In: 10th international workshop on requirments engineering foundation for software quality"},{"key":"393_CR25","unstructured":"Cysneiros LM (2007) Evaluating the effectiveness of using catalogues to elicit non-functional requirements. In: Workshop em engenharia de requisitos (WER 2007). pp 107\u2013115"},{"key":"393_CR26","doi-asserted-by":"publisher","unstructured":"Nuseibeh B, Easterbrook S (2000) Requirements engineering: a roadmap. In: Proceedings of the conference on the future of software engineering. ICSE \u201900. Association for Computing Machinery, New York, NY, pp 35\u201346. https:\/\/doi.org\/10.1145\/336512.336523","DOI":"10.1145\/336512.336523"},{"key":"393_CR27","doi-asserted-by":"publisher","unstructured":"Bittner EAC, Leimeister JM (2013) Why shared understanding matters \u2013 engineering a collaboration process for shared understanding to improve collaboration effectiveness in heterogeneous teams. In: 2013 46th Hawaii international conference on system sciences. IEEE, Piscataway, pp 106\u2013114. https:\/\/doi.org\/10.1109\/HICSS.2013.608","DOI":"10.1109\/HICSS.2013.608"},{"issue":"3","key":"393_CR28","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1007\/s00766-017-0272-z","volume":"22","author":"P Ghazi","year":"2017","unstructured":"Ghazi P, Glinz M (2017) Challenges of working with artifacts in requirements engineering and software engineering. Requir Eng 22(3):359\u2013385. https:\/\/doi.org\/10.1007\/s00766-017-0272-z","journal-title":"Requir Eng"},{"key":"393_CR29","unstructured":"Boehm BW, Brown JR, Lipow M (1976) Quantitative evaluation of software quality. In: Proceedings of the 2nd international conference on software engineering. ICSE \u201976. IEEE Computer Society Press, Washington, DC, pp 592\u2013605"},{"key":"393_CR30","unstructured":"B\u00f6rger E, H\u00f6rger B, Parnas D, Rombach D (1999) Requirements capture, documentation, and validation. In: Dagstuhl seminar. Dagstuhl Seminar"},{"issue":"3","key":"393_CR31","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1007\/BF03340367","volume":"46","author":"C Rupp","year":"2009","unstructured":"Rupp C, Simon M, Hocker F (2009) Requirements engineering und management. HMD Prax Wirtsch 46(3):94\u2013103","journal-title":"HMD Prax Wirtsch"},{"key":"393_CR32","unstructured":"Serrano M, Serrano M (2013) Ubiquitous, pervasive and mobile computing: a reusable-models-based non-functional catalogue. In: Proceedings of requirements engineering@Brazil, vol. 1005. CEUR, Aachen, DE"},{"key":"393_CR33","unstructured":"Carvalho RM, Andrade RMC, Lelli V, Silva EG, de Oliveira KM (2020) What about catalogs of non-functional requirements? In: Proceedings of REFSQ-2020 workshops, vol. 2584. CEUR, Aachen, DE"},{"key":"393_CR34","doi-asserted-by":"crossref","unstructured":"Chazette L, Brunotte W, Speith T (2021) Exploring explainability: a definition, a model, and a knowledge catalogue. In: 2021 IEEE 29th international requirements engineering conference (RE). IEEE, pp 197\u2013208","DOI":"10.1109\/RE51729.2021.00025"},{"key":"393_CR35","doi-asserted-by":"publisher","first-page":"653","DOI":"10.1016\/B978-044481862-1.50093-5","volume-title":"Handbook of human-computer interaction","author":"D Wixon","year":"1997","unstructured":"Wixon D, Wilson C (1997) Chapter 27 - the usability engineering framework for product design and evaluation. In: Helander MG, Landauer TK, Prabhu PV (eds) Handbook of human-computer interaction, 2nd edn. North-Holland, Amsterdam, pp 653\u2013688. https:\/\/doi.org\/10.1016\/B978-044481862-1.50093-5","edition":"2"},{"key":"393_CR36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-1-84996-405-0_1","volume-title":"Introduction","author":"E Hull","year":"2011","unstructured":"Hull E, Jackson K, Dick J (2011) Introduction. Springer, London, pp 1\u201323. https:\/\/doi.org\/10.1007\/978-1-84996-405-0_1"},{"key":"393_CR37","unstructured":"ISO Central Secretary: ISO\/IEC 25010:2011 systems and software engineering-systems and software quality requirements and evaluation (SQuaRE) - system and software quality models. Standard ISO\/IEC 25010:2011, International Organization for Standardization (2011). https:\/\/www.iso.org\/standard\/35733.html"},{"key":"393_CR38","unstructured":"MacKenzie CM, Laskey K, McCabe F, Brown PF, Metz R, Hamilton BA (2006) Reference model for service oriented architecture 1.0. OASIS Stand 12(S 18)"},{"issue":"1","key":"393_CR39","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1007\/BF02683509","volume":"1","author":"P Fettke","year":"2003","unstructured":"Fettke P, Loos P (2003) Classification of reference models: a methodology and its application. IseB 1(1):35\u201353. https:\/\/doi.org\/10.1007\/BF02683509","journal-title":"IseB"},{"key":"393_CR40","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/978-3-319-05152-9_2","volume-title":"OSI model","author":"MM Alani","year":"2014","unstructured":"Alani MM (2014) OSI model. Springer, Cham, pp 5\u201317. https:\/\/doi.org\/10.1007\/978-3-319-05152-9_2"},{"key":"393_CR41","doi-asserted-by":"publisher","first-page":"402","DOI":"10.1007\/11569596_43","volume-title":"Computer and information sciences - ISCIS 2005","author":"KC Weber","year":"2005","unstructured":"Weber KC, Ara\u00fajo EER, da Rocha ARC, Machado CAF, Scalet D, Salviano CF (2005) Brazilian software process reference model and assessment method. In: Yolum P, G\u00fcng\u00f6r T, G\u00fcrgen F, \u00d6zturan C (eds) Computer and information sciences - ISCIS 2005. Springer, Berlin, Heidelberg, pp 402\u2013411"},{"key":"393_CR42","doi-asserted-by":"publisher","unstructured":"Cherdantseva Y, Hilton J (2013) A reference model of information assurance and security. In: 2013 international conference on availability, reliability and security. pp 546\u2013555. https:\/\/doi.org\/10.1109\/ARES.2013.72","DOI":"10.1109\/ARES.2013.72"},{"issue":"4","key":"393_CR43","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1109\/MSP.2005.104","volume":"3","author":"P Gutmann","year":"2005","unstructured":"Gutmann P, Grigg I (2005) Security usability. IEEE Secur Priv 3(4):56\u201358. https:\/\/doi.org\/10.1109\/MSP.2005.104","journal-title":"IEEE Secur Priv"},{"issue":"02","key":"393_CR44","first-page":"102","volume":"14","author":"RC Torres","year":"2018","unstructured":"Torres RC, Martins LEG (2018) NFR catalogues for RFID middleware. J Comput Sci Technol 14(02):102\u2013108","journal-title":"J Comput Sci Technol"},{"key":"393_CR45","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1109\/RE48521.2020.00022","volume-title":"28th IEEE international requirements engineering conference (RE)","author":"RM Carvalho","year":"2020","unstructured":"Carvalho RM, Andrade RMC, Oliveira KM (2020) How developers believe invisibility impacts NFRs related to user interaction. 28th IEEE international requirements engineering conference (RE). IEEE, New York, NY, pp 102\u2013112. https:\/\/doi.org\/10.1109\/RE48521.2020.00022"},{"key":"393_CR46","doi-asserted-by":"publisher","unstructured":"Speith T (2022) A review of taxonomies of explainable artificial intelligence (xai) methods. In: Proceedings of the 2022 conference on fairness, accountability, and transparency. FAccT \u201922. Association for Computing Machinery, New York, NY. https:\/\/doi.org\/10.1145\/3531146.3534639","DOI":"10.1145\/3531146.3534639"},{"issue":"3\u20135","key":"393_CR47","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1007\/s11257-017-9195-0","volume":"27","author":"I Nunes","year":"2017","unstructured":"Nunes I, Jannach D (2017) A systematic review and taxonomy of explanations in decision support and recommender systems. User Model User-Adap Inter 27(3\u20135):393\u2013444. https:\/\/doi.org\/10.1007\/s11257-017-9195-0","journal-title":"User Model User-Adap Inter"},{"key":"393_CR48","doi-asserted-by":"publisher","unstructured":"Anjomshoae S, Najjar A, Calvaresi D, Fr\u00e4mling K (2019) Explainable agents and robots: results from a systematic literature review. In: Proceedings of the 18th international conference on autonomous agents and multiagent systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems, Richland County, SC, pp 1078\u20131088. https:\/\/doi.org\/10.5555\/3306127.3331806","DOI":"10.5555\/3306127.3331806"},{"key":"393_CR49","doi-asserted-by":"publisher","unstructured":"Abdul A, Vermeulen J, Wang D, Lim BY, Kankanhalli M (2018) Trends and trajectories for explainable, accountable and intelligible systems: an HCI research agenda. In: Proceedings of the 2018 conference on human factors in computing systems (CHI). ACM, New York, NY, pp 1\u201318. https:\/\/doi.org\/10.1145\/3173574.3174156","DOI":"10.1145\/3173574.3174156"},{"key":"393_CR50","doi-asserted-by":"publisher","unstructured":"Mathews SM (2019) Explainable artificial intelligence applications in NLP, biomedical, and malware classification: a literature review. In: Intelligent computing \u2013 proceedings of the computing conference. Springer, Cham, pp. 1269\u20131292. https:\/\/doi.org\/10.1007\/978-3-030-22868-2_90","DOI":"10.1007\/978-3-030-22868-2_90"},{"key":"393_CR51","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"AB Arrieta","year":"2020","unstructured":"Arrieta AB, D\u00edaz-Rodr\u00edguez N, Ser JD, Bennetot A, Tabik S, Barbado A, Garcia S, Gil-Lopez S, Molina D, Benjamins R, Chatila R, Herrera F (2020) Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 58:82\u2013115. https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012","journal-title":"Inf Fusion"},{"key":"393_CR52","doi-asserted-by":"publisher","unstructured":"Clinciu M-A, Hastie H (2019) A survey of explainable ai terminology. In: Alonso JM, Catala A (eds) Proceedings of the 1st workshop on interactive natural language technology for explainable artificial intelligence (NL4XAI). Association for Computational Linguistics, Stroudsburg, PA, pp 8\u201313. https:\/\/doi.org\/10.18653\/v1\/W19-8403","DOI":"10.18653\/v1\/W19-8403"},{"key":"393_CR53","doi-asserted-by":"crossref","unstructured":"Graziani M, Dutkiewicz L, Calvaresi D, Amorim JP, Yordanova K, Vered M, Nair R, Abreu PH, Blanke T, Pulignano V, Prior JO, Lauwaert L, Reijers W, Depeursinge A, Andrearcyk V, M\u00fcller H (2022) A global taxonomy of interpretable AI: unifying the terminology for the technical and social sciences. Artif Intell Rev 1\u201332","DOI":"10.1007\/s10462-022-10256-8"},{"key":"393_CR54","doi-asserted-by":"publisher","unstructured":"Langer M, Oster D, Speith T, Hermanns H, K\u00e4stner L, Schmidt E, Sesing A, Baum K (2021) What do we want from explainable artificial intelligence (XAI)? - a stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research. Articif Intell. https:\/\/doi.org\/10.1016\/j.artint.2021.103473","DOI":"10.1016\/j.artint.2021.103473"},{"issue":"3","key":"393_CR55","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1007\/s12599-010-0102-z","volume":"2","author":"JCS do Prado Leite","year":"2010","unstructured":"do Prado Leite JCS, Cappelli C (2010) Software transparency. Bus Inf. Syst Eng 2(3):127\u2013139. https:\/\/doi.org\/10.1007\/s12599-010-0102-z","journal-title":"Syst Eng"},{"key":"393_CR56","volume-title":"The coding manual for qualitative researchers","author":"J Salda\u00f1a","year":"2021","unstructured":"Salda\u00f1a J (2021) The coding manual for qualitative researchers. SAGE Publications, Thousand Oaks, CA"},{"key":"393_CR57","unstructured":"Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical report, Keele University"},{"key":"393_CR58","doi-asserted-by":"publisher","unstructured":"Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, New York, NY. pp 1\u201310. https:\/\/doi.org\/10.1145\/2601248.2601268","DOI":"10.1145\/2601248.2601268"},{"issue":"5","key":"393_CR59","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378\u2013382. https:\/\/doi.org\/10.1037\/h0031619","journal-title":"Psychol Bull"},{"issue":"1","key":"393_CR60","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159\u2013174. https:\/\/doi.org\/10.2307\/2529310","journal-title":"Biometrics"},{"issue":"1","key":"393_CR61","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1057\/ejis.2011.51","volume":"22","author":"JF Wolfswinkel","year":"2013","unstructured":"Wolfswinkel JF, Furtmueller E, Wilderom CPM (2013) Using grounded theory as a method for rigorously reviewing literature. Eur J Inf Syst 22(1):45\u201355. https:\/\/doi.org\/10.1057\/ejis.2011.51","journal-title":"Eur J Inf Syst"},{"key":"393_CR62","volume-title":"Rule-based expert systems: the MYCIN experiments of the stanford heuristic programming project","author":"BG Buchanan","year":"1984","unstructured":"Buchanan BG, Shortliffe EH (1984) Rule-based expert systems: the MYCIN experiments of the stanford heuristic programming project. Addison-Wesley, Boston, MA"},{"issue":"3","key":"393_CR63","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1609\/aimag.v39i3.2809","volume":"39","author":"DC Brock","year":"2018","unstructured":"Brock DC (2018) Learning from artificial intelligence\u2019s previous awakenings: the history of expert systems. AI Mag 39(3):3\u201315. https:\/\/doi.org\/10.1609\/aimag.v39i3.2809","journal-title":"AI Mag"},{"key":"393_CR64","volume-title":"Constructing grounded theory: a practical guide through qualitative analysis","author":"K Charmaz","year":"2006","unstructured":"Charmaz K (2006) Constructing grounded theory: a practical guide through qualitative analysis. SAGE Publications, Thousand Oaks, CA"},{"key":"393_CR65","volume-title":"Qualitative data analysis: an expanded sourcebook","author":"MB Miles","year":"1994","unstructured":"Miles MB, Huberman AM (1994) Qualitative data analysis: an expanded sourcebook. SAGE Publications, Thousand Oaks, CA"},{"key":"393_CR66","volume-title":"Transforming qualitative information: thematic analysis and code development","author":"RE Boyatzis","year":"1998","unstructured":"Boyatzis RE (1998) Transforming qualitative information: thematic analysis and code development. SAGE Publications, Thousand Oaks, CA"},{"key":"393_CR67","doi-asserted-by":"crossref","unstructured":"Chazette L, Brunotte W, Speith T (2022) Explainable software systems: from requirements analysis to system evaluation. Suppl Mater J Article https:\/\/figshare.com\/s\/f73d41c41345dd08cf39","DOI":"10.1007\/s00766-022-00393-5"},{"key":"393_CR68","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.inffus.2021.05.009","volume":"76","author":"G Vilone","year":"2021","unstructured":"Vilone G, Longo L (2021) Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf Fusion 76:89\u2013106. https:\/\/doi.org\/10.1016\/j.inffus.2021.05.009","journal-title":"Inf Fusion"},{"key":"393_CR69","doi-asserted-by":"publisher","DOI":"10.3390\/electronics10050593","author":"J Zhou","year":"2021","unstructured":"Zhou J, Gandomi AH, Chen F, Holzinger A (2021) Evaluating the quality of machine learning explanations: a survey on methods and metrics. Electronics. https:\/\/doi.org\/10.3390\/electronics10050593","journal-title":"Electronics"},{"key":"393_CR70","doi-asserted-by":"publisher","DOI":"10.3390\/electronics8080832","author":"DV Carvalho","year":"2019","unstructured":"Carvalho DV, Pereira EM, Cardoso JS (2019) Machine learning interpretability: a survey on methods and metrics. Electronics. https:\/\/doi.org\/10.3390\/electronics8080832","journal-title":"Electronics"},{"issue":"1","key":"393_CR71","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1007\/s10676-010-9253-3","volume":"13","author":"W Pieters","year":"2011","unstructured":"Pieters W (2011) Explanation and trust: What to tell the user in security and AI? Ethics Inf Technol 13(1):53\u201364. https:\/\/doi.org\/10.1007\/s10676-010-9253-3","journal-title":"Ethics Inf Technol"},{"issue":"4","key":"393_CR72","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/widm.1312","volume":"9","author":"A Holzinger","year":"2019","unstructured":"Holzinger A, Langs G, Denk H, Zatloukal K, M\u00fcller H (2019) Causability and explainability of artificial intelligence in medicine. Wiley Interdiscipl Rev Data Min Knowl Discov 9(4):1\u201313. https:\/\/doi.org\/10.1002\/widm.1312","journal-title":"Wiley Interdiscipl Rev Data Min Knowl Discov"},{"key":"393_CR73","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1007\/978-3-030-23528-4_25","volume-title":"International conference on human-computer interaction (HCI)","author":"J Hois","year":"2019","unstructured":"Hois J, Theofanou-Fuelbier D, Junk AJ (2019) How to achieve explainability and transparency in human AI interaction. International conference on human-computer interaction (HCI). Springer, Cham, CH, pp 177\u2013183. https:\/\/doi.org\/10.1007\/978-3-030-23528-4_25"},{"key":"393_CR74","doi-asserted-by":"publisher","unstructured":"Glass A, McGuinness DL, Wolverton M (2008) Toward establishing trust in adaptive agents. In: Proceedings of the 13th international conference on intelligent user interfaces (IUI). ACM, New York, NY, pp 227\u2013236. https:\/\/doi.org\/10.1145\/1378773.1378804","DOI":"10.1145\/1378773.1378804"},{"key":"393_CR75","doi-asserted-by":"publisher","unstructured":"Liao QV, Gruen DM, Miller S (2020) Questioning the AI: informing design practices for explainable AI user experiences. In: Proceedings of the 2020 conference on human factors in computing systems (CHI). ACM, New York, NY, pp 1\u201315. https:\/\/doi.org\/10.1145\/3313831.3376590","DOI":"10.1145\/3313831.3376590"},{"issue":"1","key":"393_CR76","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1007\/s00779-003-0253-8","volume":"8","author":"P Dourish","year":"2004","unstructured":"Dourish P (2004) What we talk about when we talk about context. Pers Ubiquit Comput 8(1):19\u201330. https:\/\/doi.org\/10.1007\/s00779-003-0253-8","journal-title":"Pers Ubiquit Comput"},{"key":"393_CR77","doi-asserted-by":"publisher","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u201cWhy should I trust you?\u201d: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, pp 1135\u20131144. https:\/\/doi.org\/10.1145\/2939672.2939778","DOI":"10.1145\/2939672.2939778"},{"key":"393_CR78","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338. https:\/\/doi.org\/10.1016\/j.artint.2018.07.007","journal-title":"Artif Intell"},{"key":"393_CR79","volume-title":"The culture of technology","author":"A Pacey","year":"1983","unstructured":"Pacey A (1983) The culture of technology. MIT Press, Cambridge, MA"},{"issue":"6","key":"393_CR80","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/s12599-012-0236-2","volume":"4","author":"T-F Kummer","year":"2012","unstructured":"Kummer T-F, Leimeister JM, Bick M (2012) On the importance of national culture for the design of information systems. Bus Inf Syst Eng 4(6):317\u2013330. https:\/\/doi.org\/10.1007\/s12599-012-0236-2","journal-title":"Bus Inf Syst Eng"},{"issue":"4","key":"393_CR81","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1108\/14720700410558862","volume":"4","author":"S Thomsen","year":"2004","unstructured":"Thomsen S (2004) Corporate values and corporate governance. Corp Gov 4(4):29\u201346. https:\/\/doi.org\/10.1108\/14720700410558862","journal-title":"Corp Gov"},{"key":"393_CR82","doi-asserted-by":"crossref","unstructured":"Chung L, Nixon BA (1995) Dealing with non-functional requirements: three experimental studies of a process-oriented approach. In: 1995 17th international conference on software engineering. IEEE, p 25","DOI":"10.1145\/225014.225017"},{"key":"393_CR83","doi-asserted-by":"publisher","unstructured":"Carvallo JP, Franch X, Quer C (2006) Managing non-technical requirements in cots components selection. In: 14th IEEE international requirements engineering conference (RE). IEEE, New York, NY, pp. 323\u2013326. https:\/\/doi.org\/10.1109\/RE.2006.40","DOI":"10.1109\/RE.2006.40"},{"key":"393_CR84","volume-title":"Growing object-oriented software. Guided by tests","author":"S Freeman","year":"2009","unstructured":"Freeman S, Pryce N (2009) Growing object-oriented software. Guided by tests. Addison-Wesley, Boston, MA"},{"key":"393_CR85","unstructured":"Cambridge dictionary: spectrum. Cambridge Dictionary (2022). https:\/\/dictionary.cambridge.org\/dictionary\/english\/spectrum Accessed 05 May 2022"},{"key":"393_CR86","volume-title":"Code complete","author":"S McConnell","year":"2004","unstructured":"McConnell S (2004) Code complete. Microsoft Press, Redmond, WA"},{"key":"393_CR87","unstructured":"Henin C, Daniel LM (2019) Towards a generic framework for black-box explanation methods. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI). pp 28\u201334"},{"issue":"4","key":"393_CR88","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3282878","volume":"9","author":"L Chen","year":"2019","unstructured":"Chen L, Yan D, Wang F (2019) User evaluations on sentiment-based recommendation explanations. ACM Trans Interact Intell Syst (TiiS) 9(4):1\u201338. https:\/\/doi.org\/10.1145\/3282878","journal-title":"ACM Trans Interact Intell Syst (TiiS)"},{"key":"393_CR89","unstructured":"Preece AD, Harborne D, Braines D, Tomsett R, Chakraborty S (2018) Stakeholders in explainable AI. CoRR arXiv:1810.00184"},{"key":"393_CR90","doi-asserted-by":"publisher","unstructured":"Weller A (2019) Transparency: motivations and challenges. In: Samek W, Montavon G, Vedaldi A, Hansen LK, M\u00fcller K-R (eds) Explainable AI: interpreting, explaining and visualizing deep learning. Springer, Cham, CH, pp 23\u201340. Chap. 2. https:\/\/doi.org\/10.1007\/978-3-030-28954-6_2","DOI":"10.1007\/978-3-030-28954-6_2"},{"issue":"2","key":"393_CR91","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/MS.2007.42","volume":"24","author":"M Glinz","year":"2007","unstructured":"Glinz M, Wieringa RJ (2007) Guest editors\u2019 introduction: stakeholders in requirements engineering. IEEE Softw 24(2):18\u201320. https:\/\/doi.org\/10.1109\/MS.2007.42","journal-title":"IEEE Softw"},{"issue":"1","key":"393_CR92","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1002\/hbe2.117","volume":"1","author":"MO Riedl","year":"2019","unstructured":"Riedl MO (2019) Human-centered artificial intelligence and machine learning. Hum Behav Emerg Technol 1(1):33\u201336. https:\/\/doi.org\/10.1002\/hbe2.117","journal-title":"Hum Behav Emerg Technol"},{"key":"393_CR93","doi-asserted-by":"publisher","unstructured":"McInerney J, Lacker B, Hansen S, Higley K, Bouchard H, Gruson A, Mehrotra R (2018) Explore, exploit, and explain: personalizing explainable recommendations with bandits. In: Proceedings of the 12th ACM conference on recommender systems (RecSys). ACM, New York, NY, pp 31\u201339. https:\/\/doi.org\/10.1145\/3240323.3240354","DOI":"10.1145\/3240323.3240354"},{"key":"393_CR94","doi-asserted-by":"publisher","unstructured":"Cai CJ, Jongejan J, Holbrook J (2019) The effects of example-based explanations in a machine learning interface. In: Proceedings of the 24th international conference on intelligent user interfaces (IUI). ACM, New York, NY, pp 258\u2013262. https:\/\/doi.org\/10.1145\/3301275.3302289","DOI":"10.1145\/3301275.3302289"},{"key":"393_CR95","doi-asserted-by":"publisher","unstructured":"Zanker M (2012) The influence of knowledgeable explanations on users\u2019 perception of a recommender system. In: Proceedings of the sixth ACM conference on recommender systems (RecSys). ACM, New York, NY, pp 269\u2013272. https:\/\/doi.org\/10.1145\/2365952.2366011","DOI":"10.1145\/2365952.2366011"},{"key":"393_CR96","doi-asserted-by":"publisher","unstructured":"Pu P, Chen L (2006) Trust building with explanation interfaces. In: Proceedings of the 11th international conference on intelligent user interfaces (IUI). ACM, New York, NY, pp 93\u2013100. https:\/\/doi.org\/10.1145\/1111449.1111475","DOI":"10.1145\/1111449.1111475"},{"key":"393_CR97","doi-asserted-by":"publisher","unstructured":"Kizilcec RF (2016) How much information? Effects of transparency on trust in an algorithmic interface. In: Proceedings of the 2016 conference on human factors in computing systems (CHI). ACM, New York, NY, pp 2390\u20132395. https:\/\/doi.org\/10.1145\/2858036.2858402","DOI":"10.1145\/2858036.2858402"},{"issue":"5","key":"393_CR98","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1007\/s11257-008-9051-3","volume":"18","author":"H Cramer","year":"2008","unstructured":"Cramer H, Evers V, Ramlal S, Maarten VS, Rutledge L, Stash N, Aroyo L, Wielinga B (2008) The effects of transparency on trust in and acceptance of a content-based art recommender. User Model User-Adap Inter 18(5):455. https:\/\/doi.org\/10.1007\/s11257-008-9051-3","journal-title":"User Model User-Adap Inter"},{"key":"393_CR99","doi-asserted-by":"publisher","unstructured":"Tintarev N, Masthoff J (2007) Effective explanations of recommendations: user-centered design. In: Proceedings of the 2007 ACM conference on recommender systems (RecSys). ACM, New York, NY, pp 153\u2013156. https:\/\/doi.org\/10.1145\/1297231.1297259","DOI":"10.1145\/1297231.1297259"},{"key":"393_CR100","doi-asserted-by":"publisher","unstructured":"Tsai C, Brusilovsky P (2019) Explaining recommendations in an interactive hybrid social recommender. In: Proceedings of the 24th international conference on intelligent user interfaces (IUI). ACM, New York, NY, pp 391\u2013396. https:\/\/doi.org\/10.1145\/3301275.3302318","DOI":"10.1145\/3301275.3302318"},{"issue":"4\u20135","key":"393_CR101","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1007\/s11257-011-9117-5","volume":"22","author":"N Tintarev","year":"2012","unstructured":"Tintarev N, Masthoff J (2012) Evaluating the effectiveness of explanations for recommender systems. User Model User-Adap Inter 22(4\u20135):399\u2013439. https:\/\/doi.org\/10.1007\/s11257-011-9117-5","journal-title":"User Model User-Adap Inter"},{"issue":"2","key":"393_CR102","doi-asserted-by":"publisher","first-page":"40","DOI":"10.13189\/ujca.2013.010204","volume":"1","author":"K Darlington","year":"2013","unstructured":"Darlington K (2013) Aspects of intelligent systems explanation. Univ J Control Autom 1(2):40\u201351. https:\/\/doi.org\/10.13189\/ujca.2013.010204","journal-title":"Univ J Control Autom"},{"key":"393_CR103","unstructured":"Kumar PS, Saravanan M, Suresh S (2019) Explainable classification using clustering in deep learning models. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI). pp 115\u2013121"},{"key":"393_CR104","doi-asserted-by":"publisher","unstructured":"Dodge J, Liao QV, Zhang Y, Bellamy RKE, Dugan C (2019) Explaining models: an empirical study of how explanations impact fairness judgment. In: Proceedings of the 24th international conference on intelligent user interfaces (IUI). ACM, New York, NY, pp 275\u2013285. https:\/\/doi.org\/10.1145\/3301275.3302310","DOI":"10.1145\/3301275.3302310"},{"key":"393_CR105","unstructured":"Putnam V, Conati C (2019) Exploring the need for explainable artificial intelligence (XAI) in intelligent tutoring systems (ITS). In: Joint proceedings of the ACM IUI 2019 workshops. CEUR, Aachen, DE"},{"key":"393_CR106","unstructured":"Schneider J, Handali J (2019) Personalized explanation in machine learning: a conceptualization. In: Proceedings of the 27th European conference on information systems (ECIS)"},{"key":"393_CR107","unstructured":"Gilpin LH, Testart C, Fruchter N, Adebayo J (2018) Explaining explanations to society. In: NIPS workshop on ethical, social and governance issues in AI. pp 1\u20136"},{"key":"393_CR108","unstructured":"Monteath I, Sheh R (2018) Assisted and incremental medical diagnosis using explainable artificial intelligence. In: Proceedings of the IJCAI\/ECAI workshop on explainable artificial intelligence (XAI). pp 104\u2013108"},{"key":"393_CR109","doi-asserted-by":"publisher","unstructured":"Binns R, Van\u00a0Kleek M, Veale M, Lyngs U, Zhao J, Shadbolt N (2018) \u2019It\u2019s reducing a human being to a percentage\u2019: perceptions of justice in algorithmic decisions. In: Proceedings of the 2018 conference on human factors in computing systems (CHI). ACM, New York, NY, pp. 1\u201314. https:\/\/doi.org\/10.1145\/3173574.3173951","DOI":"10.1145\/3173574.3173951"},{"key":"393_CR110","unstructured":"McCarthy K, Reilly J, McGinty L, Smyth B (2004) Thinking positively-explanatory feedback for conversational recommender systems. In: Proceedings of the European conference on case-based reasoning (ECCBR) explanation workshop. pp 115\u2013124"},{"key":"393_CR111","doi-asserted-by":"crossref","unstructured":"Lage I, Lifschitz D, Doshi-Velez F, Amir O (2019) Exploring computational user models for agent policy summarization. In Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI). pp 59\u201365","DOI":"10.24963\/ijcai.2019\/194"},{"key":"393_CR112","unstructured":"Borgo R, Cashmore M, Magazzeni D (2018) Towards providing explanations for AI planner decisions. In: Proceedings of the IJCAI\/ECAI workshop on explainable artificial intelligence (XAI). pp 11\u201317"},{"issue":"5","key":"393_CR113","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2019","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D (2019) A survey of methods for explaining black box models. ACM Comput Surv 51(5):1\u201342. https:\/\/doi.org\/10.1145\/3236009","journal-title":"ACM Comput Surv"},{"key":"393_CR114","doi-asserted-by":"publisher","unstructured":"Hohman F, Head A, Caruana R, DeLine R, Drucker SM (2019) Gamut: a design probe to understand how data scientists understand machine learning models. In: Proceedings of the 2019 chi conference on human factors in computing systems. ACM, New York, NY, pp 1\u201313. https:\/\/doi.org\/10.1145\/3290605.3300809","DOI":"10.1145\/3290605.3300809"},{"key":"393_CR115","unstructured":"Alexander IF (2004) A better fit - characterising the stakeholders. In: Grundspenkis J, Kirikova M (eds) CAiSE\u201904 workshops in connection with the 16th conference on advanced information systems engineering, Riga, Latvia, 7\u201311 June, 2004, knowledge and model driven information systems engineering for networked organisations, proceedings, vol 2. Riga Technical University, Riga, Latvia, Faculty of Computer Science and Information Technology, pp 215\u2013223"},{"key":"393_CR116","unstructured":"Zhou J, Chen F (2019) Towards trustworthy human-AI teaming under uncertainty. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI). pp 143\u2013147"},{"key":"393_CR117","unstructured":"Chen J, L\u00e9cu\u00e9 F, Pan JZ, Horrocks I, Chen H (2018) Knowledge-based transfer learning explanation. In: Proceedings of the sixteenth international conference for principles of knowledge representation and reasoning (KR). AAAI, Palo Alto, CA, pp 349\u2013358"},{"key":"393_CR118","doi-asserted-by":"publisher","unstructured":"Jacovi A, Marasovi\u0107 A, Miller T, Goldberg Y (2021) Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in AI. In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (FAccT). Association for Computing Machinery, New York, NY, pp 624\u2013635. https:\/\/doi.org\/10.1145\/3442188.3445923","DOI":"10.1145\/3442188.3445923"},{"key":"393_CR119","doi-asserted-by":"publisher","unstructured":"Nascimento N, Alencar P, Cowan D, Lucena C (2020) A reference model for iot embodied agents controlled by neural networks. In: 2020 IEEE international conference on big data (Big Data). pp 3500\u20133505. https:\/\/doi.org\/10.1109\/BigData50022.2020.9377936","DOI":"10.1109\/BigData50022.2020.9377936"},{"key":"393_CR120","unstructured":"Schneider K (2012) Abenteuer softwarequalit\u00e4t: grundlagen und verfahren f\u00fcr qualit\u00e4tssicherung und qualit\u00e4tsmanagement. dpunkt.verlag, Heidelberg, DE"},{"key":"393_CR121","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1016\/j.infsof.2015.02.009","volume":"62","author":"S Wagner","year":"2015","unstructured":"Wagner S, Goeb A, Heinemann L, Kl\u00e4s M, Lampasona C, Lochmann K, Mayr A, Pl\u00f6sch R, Seidl A, Streit J, Trendowicz A (2015) Operationalised product quality models and assessment: the quamoco approach. Inf Softw Technol 62:101\u2013123. https:\/\/doi.org\/10.1016\/j.infsof.2015.02.009","journal-title":"Inf Softw Technol"},{"key":"393_CR122","first-page":"56","volume":"1","author":"M Glinz","year":"2017","unstructured":"Glinz M (2017) A glossary of requirements engineering terminology. Stand Gloss Certif Prof Requir Eng (CPRE) Stud Exam Version 1:56","journal-title":"Stand Gloss Certif Prof Requir Eng (CPRE) Stud Exam Version"},{"key":"393_CR123","doi-asserted-by":"publisher","unstructured":"Chazette L, Kl\u00fcnder J, Balci M, Schneider K (2022) How can we develop explainable systems? insights from a literature review and an interview study. In: Proceedings of the international conference on software and system processes and international conference on global software engineering (ICSSP\u201922). ICSSP \u201922. Association for Computing Machinery, New York, NY. https:\/\/doi.org\/10.1145\/3529320.3529321","DOI":"10.1145\/3529320.3529321"},{"key":"393_CR124","unstructured":"Miller T, Howe P, Sonenberg L (2017) Explainable AI: beware of inmates running the asylum. or: how i learnt to stop worrying and love the social and behavioural sciences. In: Aha DW, Darrell T, Pazzani M, Reid D, Sammut C, Stone P (eds) Proceedings of the IJCAI 2017 workshop on explainable artificial intelligence (XAI). IJCAI, Santa Clara County, CA, pp 36\u201342"},{"key":"393_CR125","unstructured":"Hall M, Harborne D, Tomsett R, Galetic V, Quintana-Amate S, Nottle A, Preece A (2019) A systematic method to understand requirements for explainable AI (XAI) systems. In: Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence (XAI), pp 21\u201327"},{"key":"393_CR126","doi-asserted-by":"publisher","unstructured":"Dam HK, Tran T, Ghose A (2018) Explainable software analytics. In: Proceedings of the 40th international conference on software engineering: new ideas and emerging results (ICSE-NIER). Association for Computing Machinery, New York, NY, pp 53\u201356. https:\/\/doi.org\/10.1145\/3183399.3183424","DOI":"10.1145\/3183399.3183424"},{"issue":"4","key":"393_CR127","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1007\/s10699-005-5357-3","volume":"10","author":"E Weber","year":"2005","unstructured":"Weber E, Van Bouwel J, Vanderbeeken R (2005) Forms of causal explanation. Found Sci 10(4):437\u2013454","journal-title":"Found Sci"},{"issue":"4","key":"393_CR128","doi-asserted-by":"publisher","first-page":"889","DOI":"10.1093\/bjps\/axi148","volume":"56","author":"JY Halpern","year":"2005","unstructured":"Halpern JY, Pearl J (2005) Causes and explanations: a structural-model approach. Part ii: explanations. Br J Philos Sci 56(4):889\u2013911","journal-title":"Br J Philos Sci"},{"key":"393_CR129","doi-asserted-by":"crossref","unstructured":"Byrne RM (2019) Counterfactuals in explainable artificial intelligence (xai): evidence from human reasoning. In: IJCAI. pp 6276\u20136282","DOI":"10.24963\/ijcai.2019\/876"},{"issue":"1","key":"393_CR130","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BF00974201","volume":"4","author":"W Bechtel","year":"1994","unstructured":"Bechtel W (1994) Levels of description and explanation in cognitive science. Mind Mach 4(1):1\u201325","journal-title":"Mind Mach"},{"issue":"2","key":"393_CR131","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1016\/j.shpsc.2005.03.010","volume":"36","author":"W Bechtel","year":"2005","unstructured":"Bechtel W, Abrahamsen A (2005) Explanation: a mechanist alternative. Stud History Philos Sci Part C Stud History Philos Biol Biomed Sci 36(2):421\u2013441. https:\/\/doi.org\/10.1016\/j.shpsc.2005.03.010 (Mechanisms in Biology)","journal-title":"Stud History Philos Sci Part C Stud History Philos Biol Biomed Sci"},{"key":"393_CR132","unstructured":"Brinton C (2017) A framework for explanation of machine learning decisions. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI). pp 14\u201318"},{"issue":"4","key":"393_CR133","doi-asserted-by":"publisher","first-page":"497","DOI":"10.2307\/249487","volume":"23","author":"S Gregor","year":"1999","unstructured":"Gregor S, Benbasat I (1999) Explanations from intelligent systems: theoretical foundations and implications for practice. MIS Q 23(4):497\u2013530. https:\/\/doi.org\/10.2307\/249487","journal-title":"MIS Q"},{"key":"393_CR134","unstructured":"Doran D, Schulz S, Besold TR (2017) What does explainable AI really mean? A new conceptualization of perspectives. In: Proceedings of the first international workshop on comprehensibility and explanation in AI and ML, vol 2071. CEUR, Aachen, DE"},{"key":"393_CR135","unstructured":"Kim B, Wattenberg M, Gilmer J, Cai CJ, Wexler J, Vi\u00e9gas FB, Sayres R (2018) Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In: Dy JG, Krause A (eds) Proceedings of the 35th international conference on machine learning. ICML 2018. Microtome Publishing, Brookline, MA, pp 2668\u20132677. http:\/\/proceedings.mlr.press\/v80\/kim18d.html"},{"key":"393_CR136","doi-asserted-by":"crossref","unstructured":"Speith T (2021) How to evaluate explainability \u2013 a case for three criteria. In: 30th IEEE international requirements engineering conference workshops. REW 2022. IEEE, Piscataway, NJ, USA","DOI":"10.1109\/REW56159.2022.00024"},{"key":"393_CR137","unstructured":"Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. CoRR arxiv:1702.08608"},{"issue":"3","key":"393_CR138","first-page":"90","volume":"32","author":"G Friedrich","year":"2011","unstructured":"Friedrich G, Zanker M (2011) A taxonomy for generating explanations in recommender systems. AI Mag 32(3):90\u201398","journal-title":"AI Mag"},{"key":"393_CR139","unstructured":"Eiter T, Saribatur ZG, Sch\u00fcller P (2019) Abstraction for zooming-in to unsolvability reasons of grid-cell problems. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI 2019). pp 7\u201313"},{"key":"393_CR140","doi-asserted-by":"publisher","unstructured":"Vorm ES (2018) Assessing demand for transparency in intelligent systems using machine learning. In: 2018 innovations in intelligent systems and applications (INISTA). IEEE, pp. 1\u20137. https:\/\/doi.org\/10.1109\/INISTA.2018.8466328","DOI":"10.1109\/INISTA.2018.8466328"},{"key":"393_CR141","unstructured":"Juozapaitis Z, Koul A, Fern A, Erwig M, Doshi-Velez F (2019) Explainable reinforcement learning via reward decomposition. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI 2019). pp 47\u201353"},{"key":"393_CR142","unstructured":"Hoffman RR, Mueller ST, Klein G, Litman J (2018) Challenges and prospects, metrics for explainable AI"},{"key":"393_CR143","doi-asserted-by":"publisher","unstructured":"Herlocker JL, Konstan JA, Riedl J (2000) Explaining collaborative filtering recommendations. In: Proceedings of the 2000 ACM conference on computer supported cooperative work (CSCW). ACM, New York, NY, pp 241\u2013250. https:\/\/doi.org\/10.1145\/358916.358995","DOI":"10.1145\/358916.358995"},{"key":"393_CR144","volume-title":"Real world research: a resource for users of social research methods in applied settings","author":"C Robson","year":"2016","unstructured":"Robson C, McCartan K (2016) Real world research: a resource for users of social research methods in applied settings, 4th edn. Wiley, Chichester","edition":"4"},{"key":"393_CR145","doi-asserted-by":"crossref","unstructured":"Ehsan U, Tambwekar P, Chan L, Harrison B, Riedl MO (2019) Automated rationale generation: a technique for explainable ai and its effects on human perceptions. In: Proceedings of the 24th international conference on intelligent user interfaces. pp 263\u2013274","DOI":"10.1145\/3301275.3302316"},{"key":"393_CR146","unstructured":"Olson ML, Neal L, Li F, Wong W-K (2019) Counterfactual states for atari agents via generative deep learning. In: Proceedings of the IJCAI workshop on explainable artificial intelligence (XAI 2019). pp 87\u201393"},{"key":"393_CR147","doi-asserted-by":"publisher","first-page":"720","DOI":"10.2197\/ipsjjip.27.720","volume":"27","author":"M Sato","year":"2019","unstructured":"Sato M, Nagatani K, Sonoda T, Zhang Q, Ohkuma T (2019) Context style explanation for recommender systems. J Inf Process 27:720\u2013729. https:\/\/doi.org\/10.2197\/ipsjjip.27.720","journal-title":"J Inf Process"},{"key":"393_CR148","doi-asserted-by":"crossref","unstructured":"Wang N, Wang H, Jia Y, Yin Y (2018) Explainable recommendation via multi-task learning in opinionated text data. In: The 41st international ACM SIGIR conference on research & development in information retrieval. pp 165\u2013174","DOI":"10.1145\/3209978.3210010"},{"key":"393_CR149","doi-asserted-by":"publisher","unstructured":"Aydemir FB, Dalpiaz F (2018) A roadmap for ethics-aware software engineering. In: Proceedings of the international workshop on software fairness (FairWare). ACM, New York, NY, pp 15\u201321. https:\/\/doi.org\/10.1145\/3194770.3194778","DOI":"10.1145\/3194770.3194778"}],"container-title":["Requirements Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00766-022-00393-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00766-022-00393-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00766-022-00393-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T14:32:07Z","timestamp":1669300327000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00766-022-00393-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,14]]},"references-count":149,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["393"],"URL":"https:\/\/doi.org\/10.1007\/s00766-022-00393-5","relation":{},"ISSN":["0947-3602","1432-010X"],"issn-type":[{"value":"0947-3602","type":"print"},{"value":"1432-010X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,14]]},"assertion":[{"value":"31 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 November 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}