{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,14]],"date-time":"2026-06-14T06:34:08Z","timestamp":1781418848547,"version":"3.54.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T00:00:00Z","timestamp":1730419200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T00:00:00Z","timestamp":1730419200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/21iv005e","name":"German Federal Ministry of Education and Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/21iv005e","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009534","name":"Universit\u00e4t Stuttgart","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100009534","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Explainable artificial intelligence (XAI) is a field of study that focuses on the development process of AI-based systems while making their decision-making processes understandable and transparent for users. Research already identified explainability as an emerging requirement for AI-based systems that use machine learning (ML) techniques. However, there is a notable absence of studies investigating how ML practitioners perceive the concept of explainability, the challenges they encounter, and the potential trade-offs with other quality attributes. In this study, we want to discover how practitioners define explainability for AI-based systems and what challenges they encounter in making them explainable. Furthermore, we explore how explainability interacts with other quality attributes. To this end, we conducted semi-structured interviews with 14 ML practitioners from 11 companies. Our study reveals diverse viewpoints on explainability and applied practices. Results suggest that the importance of explainability lies in enhancing transparency, refining models, and mitigating bias. Methods like SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanation (LIME) are frequently used by ML practitioners to understand how models work, while tailored approaches are typically adopted to meet the specific requirements of stakeholders. Moreover, we have discerned emerging challenges in eight categories. Issues such as effective communication with non-technical stakeholders and the absence of standardized approaches are frequently stated as recurring hurdles. We contextualize these findings in terms of requirements engineering and conclude that industry currently lacks a standardized framework to address arising explainability needs.<\/jats:p>","DOI":"10.1007\/s10664-024-10565-2","type":"journal-article","created":{"date-parts":[[2024,11,1]],"date-time":"2024-11-01T06:04:39Z","timestamp":1730441079000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["How do ML practitioners perceive explainability? an interview study of practices and challenges"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8953-9624","authenticated-orcid":false,"given":"Umm-e-","family":"Habiba","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1272-9873","authenticated-orcid":false,"given":"Mohammad Kasra","family":"Habib","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5788-0991","authenticated-orcid":false,"given":"Justus","family":"Bogner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6121-2731","authenticated-orcid":false,"given":"Jonas","family":"Fritzsch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5256-8429","authenticated-orcid":false,"given":"Stefan","family":"Wagner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,11,1]]},"reference":[{"key":"10565_CR1","unstructured":"Atakishiyev S, Salameh M, Yao H, Goebel R (2021) Explainable artificial intelligence for autonomous driving: a comprehensive overview and field guide for future research directions. arXiv:2112.11561"},{"issue":"4","key":"10565_CR2","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1007\/s10664-021-10072-8","volume":"27","author":"S Baltes","year":"2022","unstructured":"Baltes S, Ralph P (2022) Sampling in software engineering research: a critical review and guidelines. Empir Softw Eng 27(4):94","journal-title":"Empir Softw Eng"},{"key":"10565_CR3","doi-asserted-by":"crossref","unstructured":"Brennen A (2020) What do people really want when they say they want\u201c explainable ai?\" we asked 60 stakeholders. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems, pp 1\u20137","DOI":"10.1145\/3334480.3383047"},{"key":"10565_CR4","doi-asserted-by":"crossref","unstructured":"Brunotte W, Chazette L, Korte K (2021) Can explanations support privacy awareness? a research roadmap. In: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), pp 176\u2013180. IEEE","DOI":"10.1109\/REW53955.2021.00032"},{"key":"10565_CR5","doi-asserted-by":"crossref","unstructured":"Caruana R, Lou Y, Gehrke J, Koch P, Sturm M, Elhadad N (2015) Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp 1721\u20131730","DOI":"10.1145\/2783258.2788613"},{"issue":"8","key":"10565_CR6","doi-asserted-by":"publisher","first-page":"832","DOI":"10.3390\/electronics8080832","volume":"8","author":"DV Carvalho","year":"2019","unstructured":"Carvalho DV, Pereira EM, Cardoso JS (2019) Machine learning interpretability: a survey on methods and metrics. Electron 8(8):832","journal-title":"Electron"},{"issue":"4","key":"10565_CR7","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1007\/s00766-020-00333-1","volume":"25","author":"L Chazette","year":"2020","unstructured":"Chazette L, Schneider K (2020) Explainability as a non-functional requirement: challenges and recommendations. Requir Eng 25(4):493\u2013514","journal-title":"Requir Eng"},{"key":"10565_CR8","doi-asserted-by":"crossref","unstructured":"Chazette L, Brunotte W, Speith T (2021) Exploring explainability: a definition, a model, and a knowledge catalogue. In: 2021 IEEE 29th International Requirements Engineering Conference (RE), pp 197\u2013208. IEEE","DOI":"10.1109\/RE51729.2021.00025"},{"key":"10565_CR9","first-page":"1591","volume":"2021","author":"S Dhanorkar","year":"2021","unstructured":"Dhanorkar S, Wolf CT, Qian K, Xu A, Popa L, Li Y (2021) Who needs to know what, when?: broadening the explainable ai (xai) design space by looking at explanations across the ai lifecycle. Designing Interactive Systems Conference 2021:1591\u20131602","journal-title":"Designing Interactive Systems Conference"},{"key":"10565_CR10","unstructured":"Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. arXiv:1702.08608"},{"issue":"9","key":"10565_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3561048","volume":"55","author":"R Dwivedi","year":"2023","unstructured":"Dwivedi R, Dave D, Naik H, Singhal S, Omer R, Patel P, Qian B, Wen Z, Shah T, Morgan G et al (2023) Explainable ai (xai): core ideas, techniques, and solutions. ACM Comput Surv 55(9):1\u201333","journal-title":"ACM Comput Surv"},{"key":"10565_CR12","unstructured":"European Parliament, Council of the European Union: Regulation (EU) 2016\/679 of the European Parliament and of the Council. https:\/\/data.europa.eu\/eli\/reg\/2016\/679\/oj Accessed 2024-07-20"},{"key":"10565_CR13","doi-asserted-by":"crossref","unstructured":"Gilpin LH, Bau D, Yuan BZ, Bajwa A, Specter M, Kagal L (2018) Explaining explanations: an overview of interpretability of machine learning.In: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp 80\u201389. IEEE","DOI":"10.1109\/DSAA.2018.00018"},{"key":"10565_CR14","doi-asserted-by":"crossref","unstructured":"Habiba UE, Bogner J, Wagner S (2022) Can requirements engineering support explainable artificial intelligence? towards a user-centric approach for explainability requirements. In: 2022 IEEE 30th International Requirements Engineering Conference Workshops (REW), pp 162\u2013165","DOI":"10.1109\/REW56159.2022.00038"},{"key":"10565_CR15","doi-asserted-by":"crossref","unstructured":"Henin C, Le\u00a0M\u00e9tayer D (2021) A multi-layered approach for tailored black-box explanations. In: International conference on pattern recognition, pp 5\u201319. Springer","DOI":"10.1007\/978-3-030-68796-0_1"},{"key":"10565_CR16","doi-asserted-by":"publisher","first-page":"1117848","DOI":"10.3389\/fcomp.2023.1117848","volume":"5","author":"RR Hoffman","year":"2023","unstructured":"Hoffman RR, Mueller ST, Klein G, Jalaeian M, Tate C (2023) Explainable ai: roles and stakeholders, desirements and challenges. Front Comput Sci 5:1117848","journal-title":"Front Comput Sci"},{"key":"10565_CR17","doi-asserted-by":"crossref","unstructured":"Ishikawa F, Matsuno Y (2020) Evidence-driven requirements engineering for uncertainty of machine learning-based systems. In: 2020 IEEE 28th International Requirements Engineering Conference (RE), pp 346\u2013351. IEEE","DOI":"10.1109\/RE48521.2020.00046"},{"key":"10565_CR18","unstructured":"Jansen\u00a0Ferreira J, Monteiro M (2021) Designer-user communication for xai: an epistemological approach to discuss xai design. arXiv e-prints, 2105"},{"issue":"2","key":"10565_CR19","doi-asserted-by":"publisher","first-page":"021002","DOI":"10.1088\/1741-2552\/ab8131","volume":"17","author":"W Jin","year":"2020","unstructured":"Jin W, Fatehi M, Abhishek K, Mallya M, Toyota B, Hamarneh G (2020) Artificial intelligence in glioma imaging: challenges and advances. J Neural Eng 17(2):021002","journal-title":"J Neural Eng"},{"key":"10565_CR20","unstructured":"Jin W, Fan J, Gromala D, Pasquier P, Hamarneh G (2023) Invisible users: uncovering end-users\u2019 requirements for explainable ai via explanation forms and goals. arXiv:2302.06609"},{"key":"10565_CR21","doi-asserted-by":"crossref","unstructured":"K\u00e4stner L, Langer M, Lazar V, Schom\u00e4cker A, Speith T, Sterz S (2021) On the relation of trust and explainability: Why to engineer for trustworthiness. In: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), pp 169\u2013175. IEEE","DOI":"10.1109\/REW53955.2021.00031"},{"key":"10565_CR22","doi-asserted-by":"crossref","unstructured":"K\u00f6hl MA, Baum K, Langer M, Oster D, Speith T, Bohlender D (2019) Explainability as a non-functional requirement. In: 2019 IEEE 27th International Requirements Engineering Conference (RE), pp 363\u2013368. IEEE","DOI":"10.1109\/RE.2019.00046"},{"key":"10565_CR23","doi-asserted-by":"crossref","unstructured":"K\u00f6hl MA, Baum K, Langer M, Oster D, Speith T, Bohlender D (2019) Explainability as a non-functional requirement. In: 2019 IEEE 27th International Requirements Engineering Conference (RE), pp 363\u2013368. IEEE","DOI":"10.1109\/RE.2019.00046"},{"key":"10565_CR24","doi-asserted-by":"crossref","unstructured":"Krishna S, Han T, Gu A, Pombra J, Jabbari S, Wu S, Lakkaraju H (2022) The disagreement problem in explainable machine learning: a practitioner\u2019s perspective. arXiv:2202.01602","DOI":"10.21203\/rs.3.rs-2963888\/v1"},{"key":"10565_CR25","doi-asserted-by":"crossref","unstructured":"Kuwajima H, Ishikawa F (2019) Adapting square for quality assessment of artificial intelligence systems. In: 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp 13\u201318. IEEE","DOI":"10.1109\/ISSREW.2019.00035"},{"key":"10565_CR26","unstructured":"Lagioia F et al (2020) The impact of the general data protection regulation (gdpr) on artificial intelligence"},{"key":"10565_CR27","unstructured":"Lakkaraju H, Slack D, Chen Y, Tan C, Singh S (2022) Rethinking explainability as a dialogue: a practitioner\u2019s perspective. arXiv:2202.01875"},{"issue":"3","key":"10565_CR28","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton ZC (2018) The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16(3):31\u201357","journal-title":"Queue"},{"issue":"3","key":"10565_CR29","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton ZC (2018) The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16(3):31\u201357","journal-title":"Queue"},{"key":"10565_CR30","unstructured":"Longo L, Goebel R, Lecue F, Kieseberg P, Holzinger A (2020) Explainable artificial intelligence: Concepts, applications, research challenges and visions. In: Machine learning and knowledge extraction: 4th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2020, Dublin, Ireland, August 25\u201328, 2020, Proceedings, pp 1\u201316. Springer"},{"key":"10565_CR31","unstructured":"Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. Adv Neural Inf Process Syst 30"},{"key":"10565_CR32","doi-asserted-by":"publisher","first-page":"103655","DOI":"10.1016\/j.jbi.2020.103655","volume":"113","author":"AF Markus","year":"2021","unstructured":"Markus AF, Kors JA, Rijnbeek PR (2021) The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform 113:103655","journal-title":"J Biomed Inform"},{"key":"10565_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338","journal-title":"Artif Intell"},{"key":"10565_CR34","doi-asserted-by":"crossref","unstructured":"Montavon G, Samek W, M\u00fcller KR (2018) Methods for interpreting and understanding deep neural networks. Digit Signal Process 73","DOI":"10.1016\/j.dsp.2017.10.011"},{"key":"10565_CR35","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u2019why should i trust you?\u2019 explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135\u20131144","DOI":"10.1145\/2939672.2939778"},{"key":"10565_CR36","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","volume":"14","author":"P Runeson","year":"2009","unstructured":"Runeson P, H\u00f6st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14:131\u2013164","journal-title":"Empir Softw Eng"},{"key":"10565_CR37","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2019.113100","volume":"144","author":"S Sachan","year":"2020","unstructured":"Sachan S, Yang JB, Xu DL, Benavides DE, Li Y (2020) An explainable ai decision-support-system to automate loan underwriting. Expert Syst Appl 144:113100","journal-title":"Expert Syst Appl"},{"key":"10565_CR38","doi-asserted-by":"crossref","unstructured":"Sadeghi M, Kl\u00f6s V, Vogelsang A (2021) Cases for explainable software systems: characteristics and examples. In: 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), pp 181\u2013187. IEEE","DOI":"10.1109\/REW53955.2021.00033"},{"key":"10565_CR39","doi-asserted-by":"crossref","unstructured":"Seaman CB (2008) Qualitative methods. Guid Adv Empir Softw Eng 35\u201362","DOI":"10.1007\/978-1-84800-044-5_2"},{"key":"10565_CR40","doi-asserted-by":"crossref","unstructured":"Sheh R (2021) Explainable artificial intelligence requirements for safe, intelligent robots. In: 2021 IEEE International Conference on Intelligence and Safety for Robotics (ISR), pp 382\u2013387. IEEE","DOI":"10.1109\/ISR50024.2021.9419498"},{"key":"10565_CR41","doi-asserted-by":"crossref","unstructured":"Suresh H, Gomez SR, Nam KK, Satyanarayan A (2021) Beyond expertise and roles: a framework to characterize the stakeholders of interpretable machine learning and their needs. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp 1\u201316","DOI":"10.1145\/3411764.3445088"},{"key":"10565_CR42","doi-asserted-by":"crossref","unstructured":"Vogelsang A, Borg M (2019) Requirements engineering for machine learning: perspectives from data scientists. In: 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), pp 245\u2013251. IEEE","DOI":"10.1109\/REW.2019.00050"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10565-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10565-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10565-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,7]],"date-time":"2025-01-07T14:11:46Z","timestamp":1736259106000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10565-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,1]]},"references-count":42,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10565"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10565-2","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,1]]},"assertion":[{"value":"1 October 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 November 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declared that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}],"article-number":"18"}}