{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,2]],"date-time":"2026-08-02T11:51:31Z","timestamp":1785671491793,"version":"3.56.0"},"reference-count":70,"publisher":"Emerald","issue":"7","license":[{"start":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T00:00:00Z","timestamp":1651449600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["INTR"],"published-print":{"date-parts":[[2022,12,19]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title><jats:p>Inscrutable machine learning (ML) models are part of increasingly many information systems. Understanding how these models behave, and what their output is based on, is a challenge for developers let\u00a0alone non-technical end users.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title><jats:p>The authors investigate how AI systems and their decisions ought to be explained for end users through a systematic literature review.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Findings<\/jats:title><jats:p>The authors\u2019 synthesis of the literature suggests that AI system communication for end users has five high-level goals: (1) understandability, (2) trustworthiness, (3) transparency, (4) controllability and (5) fairness. The authors identified several design recommendations, such as offering personalized and on-demand explanations and focusing on the explainability of key functionalities instead of aiming to explain the whole system. There exists multiple trade-offs in AI system explanations, and there is no single best solution that fits all cases.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Research limitations\/implications<\/jats:title><jats:p>Based on the synthesis, the authors provide a design framework for explaining AI systems to end users. The study contributes to the work on AI governance by suggesting guidelines on how to make AI systems more understandable, fair, trustworthy, controllable and transparent.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title><jats:p>This literature review brings together the literature on AI system communication and explainable AI (XAI) for end users. Building on previous academic literature on the topic, it provides synthesized insights, design recommendations and future research agenda.<\/jats:p><\/jats:sec>","DOI":"10.1108\/intr-08-2021-0600","type":"journal-article","created":{"date-parts":[[2022,5,5]],"date-time":"2022-05-05T10:51:34Z","timestamp":1651747894000},"page":"1-31","source":"Crossref","is-referenced-by-count":128,"title":["How to explain AI systems to end users: a systematic literature review and research agenda"],"prefix":"10.1108","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4285-0073","authenticated-orcid":false,"given":"Samuli","family":"Laato","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miika","family":"Tiainen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2236-3278","authenticated-orcid":false,"given":"A.K.M.","family":"Najmul Islam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1981-566X","authenticated-orcid":false,"given":"Matti","family":"M\u00e4ntym\u00e4ki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2022,5,2]]},"reference":[{"key":"key2022121123451837500_ref001","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: a survey on explainable artificial intelligence (XAI)","volume":"6","year":"2018","journal-title":"IEEE Access"},{"key":"key2022121123451837500_ref002","first-page":"1078","article-title":"Explainable agents and robots: results from a systematic literature review","year":"2019"},{"issue":"11","key":"key2022121123451837500_ref003","doi-asserted-by":"publisher","DOI":"10.3390\/app11115088","article-title":"Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: a systematic review","volume":"11","year":"2021","journal-title":"Applied Sciences"},{"key":"key2022121123451837500_ref004","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities, and challenges toward responsible AI","volume":"58","year":"2020","journal-title":"Information Fusion"},{"issue":"4","key":"key2022121123451837500_ref005","doi-asserted-by":"publisher","first-page":"259","DOI":"10.17705\/2msqe.00037","article-title":"Challenges of explaining the behavior of blackbox AI systems","volume":"19","year":"2020","journal-title":"MIS Quarterly Executive"},{"issue":"2","key":"key2022121123451837500_ref006","doi-asserted-by":"publisher","DOI":"10.17705\/1jais.00664","article-title":"Sociotechnical envelopment of artificial intelligence: an approach to organizational deployment of inscrutable artificial intelligence systems","volume":"22","year":"2021","journal-title":"Journal of the Association for Information Systems"},{"issue":"1","key":"key2022121123451837500_ref007","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10462-018-9654-y","article-title":"A review on deep learning for recommender systems: challenges and remedies","volume":"52","year":"2019","journal-title":"Artificial Intelligence Review"},{"key":"key2022121123451837500_ref008","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3173574.3173951","article-title":"It's reducing a human being to a percentage\u201d: perceptions of justice in algorithmic decisions","year":"2018"},{"key":"key2022121123451837500_ref009","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3334480.3383047","article-title":"What do people really want when they say they want \u201cexplainable AI?\u201d We asked 60 stakeholders","year":"2020"},{"key":"key2022121123451837500_ref010","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1007\/978-3-642-16178-0_5","article-title":"Do you get it? User-evaluated explainable BDI agents","year":"2010"},{"issue":"1","key":"key2022121123451837500_ref011","doi-asserted-by":"publisher","DOI":"10.1177\/2053951715622512","article-title":"How the machine \u201cthinks\u201d: understanding opacity in machine learning algorithms","volume":"3","year":"2016","journal-title":"Big Data and Society"},{"key":"key2022121123451837500_ref012","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1109\/ICHI.2015.26","article-title":"The role of explanations on trust and reliance in clinical decision support systems","year":"2015"},{"key":"key2022121123451837500_ref013","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359206","article-title":"Hello AI\": uncovering the onboarding needs of medical practitioners for human\u2013AI collaborative decision-making","year":"2019"},{"issue":"4","key":"key2022121123451837500_ref014","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1007\/s00766-020-00333-1","article-title":"Explainability as a non-functional requirement: challenges and recommendations","volume":"25","year":"2020","journal-title":"Requirements Engineering"},{"key":"key2022121123451837500_ref015","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3290605.3300789","article-title":"Explaining decision-making algorithms through UI: strategies to help non-expert stakeholders","year":"2019"},{"key":"key2022121123451837500_ref016","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1007\/978-3-030-57321-8_18","article-title":"Scenario-based requirements elicitation for user-centric explainable AI","year":"2020"},{"key":"key2022121123451837500_ref017","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2021.102383","article-title":"Artificial intelligence in information systems research: a systematic literature review and research agenda","volume":"60","year":"2021","journal-title":"International Journal of Information Management"},{"issue":"5","key":"key2022121123451837500_ref018","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-008-9051-3","article-title":"The effects of transparency on trust in and acceptance of a content-based art recommender","volume":"18","year":"2008","journal-title":"User Modeling and User-Adapted Interaction"},{"key":"key2022121123451837500_ref019","unstructured":"Dawes, S. (2021), \u201cAn autonomous robot may have already killed people \u2013 here's how the weapons could be more destabilizing than nukes\u201d, available at: https:\/\/theconversation.com\/an-autonomous-robot-may-have-already-killed-people-heres-how-the-weapons-could-be-more-destabilizing-than-nukes-168049 (accessed 2 April 2022)."},{"key":"key2022121123451837500_ref020","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1145\/3301275.3302310","article-title":"Explaining models: an empirical study of how explanations impact fairness judgment","year":"2019"},{"key":"key2022121123451837500_ref021","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s40747-021-00424-8","article-title":"A systematic review on AI\/ML approaches against COVID-19 outbreak","year":"2021","journal-title":"Complex and Intelligent Systems"},{"key":"key2022121123451837500_ref022","doi-asserted-by":"publisher","first-page":"210","DOI":"10.23919\/MIPRO.2018.8400040","article-title":"Explainable artificial intelligence: a survey","year":"2018"},{"key":"key2022121123451837500_ref023","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1145\/3301275.3302316","article-title":"Automated rationale generation: a technique for explainable AI and its effects on human perceptions","year":"2019"},{"key":"key2022121123451837500_ref024","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1145\/3172944.3172961","article-title":"Bringing transparency design into practice","year":"2018"},{"key":"key2022121123451837500_ref025","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3173574.3174006","article-title":"Communicating algorithmic process in online behavioral advertising","year":"2018"},{"key":"key2022121123451837500_ref026","unstructured":"European Commission (2020), \u201cWhite paper on artificial intelligence\u2014a European approach to excellence and trust\u201d, available at: https:\/\/ec.europa.eu\/info\/sites\/default\/files\/commission-white-paper-artificial-intelligence-feb2020_en.pdf (accessed 28 March 2022)."},{"key":"key2022121123451837500_ref027","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103521","article-title":"Making sense of raw input","volume":"299","year":"2021","journal-title":"Artificial Intelligence"},{"key":"key2022121123451837500_ref028","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s12525-021-00493-0","article-title":"AI invading the workplace: negative emotions towards the organizational use of personal virtual assistants","year":"2021","journal-title":"Electronic Markets"},{"issue":"9","key":"key2022121123451837500_ref029","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1038\/s42256-019-0088-2","article-title":"The global landscape of AI ethics guidelines","volume":"1","year":"2019","journal-title":"Nature Machine Intelligence"},{"issue":"7","key":"key2022121123451837500_ref030","doi-asserted-by":"publisher","DOI":"10.3390\/info11070363","article-title":"Applying DevOps practices of continuous automation for machine learning","volume":"11","year":"2020","journal-title":"Information"},{"issue":"12","key":"key2022121123451837500_ref031","doi-asserted-by":"publisher","first-page":"2049","DOI":"10.1016\/j.infsof.2013.07.010","article-title":"A systematic review of systematic review process research in\u00a0software engineering","volume":"55","year":"2013","journal-title":"Information and Software Technology"},{"issue":"2","key":"key2022121123451837500_ref032","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/s13748-019-00179-x","article-title":"Improving transparency of deep neural inference process","volume":"8","year":"2019","journal-title":"Progress in Artificial Intelligence"},{"key":"key2022121123451837500_ref033","doi-asserted-by":"publisher","first-page":"609","DOI":"10.1007\/978-3-030-85447-8_50","article-title":"Digital transformation of software development: implications for the future of work","year":"2021"},{"key":"key2022121123451837500_ref034","doi-asserted-by":"publisher","DOI":"10.1007\/s10796-022-10267-4","article-title":"Trends and trajectories in the software industry: implications for the future of work","year":"2022","journal-title":"Information Systems Frontiers"},{"issue":"1","key":"key2022121123451837500_ref035","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1108\/INTR-11-2020-0668","article-title":"Artificial intelligence and robots in individuals' lives: how to align technological possibilities and ethical issues","volume":"31","year":"2021","journal-title":"Internet Research"},{"key":"key2022121123451837500_ref036","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1145\/1620545.1620576","article-title":"Assessing demand for intelligibility in context-aware applications","year":"2009"},{"key":"key2022121123451837500_ref037","doi-asserted-by":"publisher","first-page":"2119","DOI":"10.1145\/1518701.1519023","article-title":"Why and why not explanations improve the intelligibility of context-aware intelligent systems","year":"2009"},{"key":"key2022121123451837500_ref038","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s43681-022-00143-x","article-title":"Defining organizational AI governance","year":"2022","journal-title":"AI and Ethics"},{"issue":"1","key":"key2022121123451837500_ref039","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1080\/10580530.2020.1849465","article-title":"Explainable artificial intelligence: objectives, stakeholders, and future research opportunities","volume":"39","year":"2022","journal-title":"Information Systems Management"},{"issue":"1","key":"key2022121123451837500_ref040","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1109\/TVCG.2019.2934267","article-title":"ProtoSteer: steering deep sequence model with prototypes","volume":"26","year":"2019","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"key2022121123451837500_ref041","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-022-01415-0","article-title":"What about investors? ESG analyses as tools for ethics-based AI auditing","year":"2022","journal-title":"AI and Society"},{"key":"key2022121123451837500_ref042","doi-asserted-by":"publisher","DOI":"10.1007\/s10796-022-10269-2","article-title":"Co-shaping an ecosystem for responsible AI: an analysis of expectation work in response to a technological frame","year":"2022","journal-title":"Information Systems Frontiers"},{"key":"key2022121123451837500_ref043","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1145\/3287560.3287596","article-title":"Model cards for model reporting","year":"2019"},{"issue":"7","key":"key2022121123451837500_ref044","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-151-4-200908180-00135","article-title":"Preferred reporting items for systematic reviews and meta-analyzes: the PRISMA statement","volume":"6","year":"2009","journal-title":"PLoS Medicine"},{"key":"key2022121123451837500_ref045","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1016\/j.ijhcs.2017.04.005","article-title":"Gamified crowdsourcing: conceptualization, literature review, and future agenda","volume":"106","year":"2017","journal-title":"International Journal of Human-Computer Studies"},{"key":"key2022121123451837500_ref046","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1145\/3340631.3394841","article-title":"Exploring mental models for transparent and controllable recommender systems: a qualitative study","year":"2020"},{"key":"key2022121123451837500_ref047","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3173574.3174223","article-title":"I lead, you help, but only with enough details: understanding user experience of co-creation with artificial intelligence","year":"2018"},{"key":"key2022121123451837500_ref048","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2019.105405","article-title":"Toward safer highways, application of XGBoost and SHAP for real-time accident detection and feature analysis","volume":"136","year":"2020","journal-title":"Accident Analysis and Prevention"},{"key":"key2022121123451837500_ref049","article-title":"Exploring the need for explainable artificial intelligence (XAI) in intelligent tutoring systems (ITS)","year":"2019"},{"key":"key2022121123451837500_ref050","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/0960085X.2021.1955628","article-title":"Understanding dark side of artificial intelligence (AI) integrated business analytics: assessing firm's operational inefficiency and competitiveness","year":"2021","journal-title":"European Journal of Information Systems"},{"key":"key2022121123451837500_ref051","doi-asserted-by":"publisher","first-page":"1135","DOI":"10.1145\/2939672.2939778","article-title":"\u201cWhy should I trust you?\u201d Explaining the predictions of any classifier","year":"2016"},{"key":"key2022121123451837500_ref052","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1007\/978-3-030-50334-5_8","article-title":"Color for characters-effects of visual explanations of AI on trust and observability","year":"2020"},{"key":"key2022121123451837500_ref053","article-title":"From ethical principles to governed AI","year":"2021"},{"issue":"4","key":"key2022121123451837500_ref054","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3419764","article-title":"Bridging the gap between ethics and practice: guidelines for reliable, safe, and trustworthy human-centered AI systems","volume":"10","year":"2020","journal-title":"ACM Transactions on Interactive Intelligent Systems"},{"key":"key2022121123451837500_ref055","unstructured":"Smith, G. (2021), \u201cAn epic failure: overstated AI claims in medicine - independent investigations are finding that AI algorithms used in hospitals are not all they claim to be\u201d, available at: https:\/\/mindmatters.ai\/2021\/08\/an-epic-failure-overstated-ai-claims-in-medicine (accessed 2 April 2022)."},{"key":"key2022121123451837500_ref056","volume-title":"Basics of Qualitative Research Techniques","year":"1998"},{"issue":"4","key":"key2022121123451837500_ref057","first-page":"37","article-title":"Using AI to enhance business operations","volume":"60","year":"2019","journal-title":"MIT Sloan Management Review"},{"issue":"1","key":"key2022121123451837500_ref058","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1111\/isj.12253","article-title":"Explaining the link between technostress and technology addiction for social networking sites: a study of distraction as a coping behavior","volume":"30","year":"2020","journal-title":"Information Systems Journal"},{"key":"key2022121123451837500_ref059","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2020.102493","article-title":"Interpretable confidence measures for decision support systems","volume":"144","year":"2020","journal-title":"International Journal of Human\u2013Computer Studies"},{"key":"key2022121123451837500_ref060","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2020.103404","article-title":"Evaluating XAI: a comparison of rule-based and example-based explanations","volume":"291","year":"2021","journal-title":"Artificial Intelligence"},{"key":"key2022121123451837500_ref061","doi-asserted-by":"publisher","first-page":"1607","DOI":"10.1007\/s13347-021-00477-0","article-title":"Transparency and the black box problem: why we do not trust AI","volume":"34","year":"2021","journal-title":"Philosophy and Technology"},{"key":"key2022121123451837500_ref062","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3290605.3300831","article-title":"Designing theory-driven user-centric explainable AI","year":"2019"},{"key":"key2022121123451837500_ref063","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/s12193-020-00332-0","article-title":"\u201cLet me explain!\u201d: exploring the potential of virtual agents in explainable AI interaction design","volume":"15","year":"2019","journal-title":"Journal on Multimodal User Interfaces"},{"key":"key2022121123451837500_ref064","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/3308532.3329441","article-title":"Do you trust me?\" Increasing user-trust by integrating virtual agents in explainable AI interaction design","year":"2019"},{"key":"key2022121123451837500_ref065","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1007\/978-3-030-87626-5_19","article-title":"Demystifying artificial intelligence for end-users: findings from a participatory machine learning show","year":"2021"},{"key":"key2022121123451837500_ref066","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2601248.2601268","article-title":"Guidelines for snowballing in systematic literature studies and a replication in software engineering","year":"2014"},{"key":"key2022121123451837500_ref067","doi-asserted-by":"publisher","first-page":"1","DOI":"10.48550\/arXiv.1902.06019","article-title":"Outlining the design space of explainable intelligent systems for medical diagnosis","year":"2019"},{"key":"key2022121123451837500_ref068","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3290605.3300509","article-title":"Understanding the effect of accuracy on trust in machine learning models","year":"2019"},{"issue":"1","key":"key2022121123451837500_ref069","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1080\/15332861.2020.1832817","article-title":"Evaluating if trust and personal information privacy concerns are barriers to using health insurance that explicitly utilizes AI","volume":"20","year":"2020","journal-title":"Journal of Internet Commerce"},{"key":"key2022121123451837500_ref070","doi-asserted-by":"publisher","article-title":"Explainable AI for designers: a human-centered perspective on mixed-initiative co-creation","year":"2018","DOI":"10.1109\/CIG.2018.8490433"}],"container-title":["Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/INTR-08-2021-0600\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/INTR-08-2021-0600\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T21:36:40Z","timestamp":1753393000000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/intr\/article\/32\/7\/1-31\/187546"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,2]]},"references-count":70,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2022,5,2]]},"published-print":{"date-parts":[[2022,12,19]]}},"alternative-id":["10.1108\/INTR-08-2021-0600"],"URL":"https:\/\/doi.org\/10.1108\/intr-08-2021-0600","relation":{},"ISSN":["1066-2243"],"issn-type":[{"value":"1066-2243","type":"print"}],"subject":[],"published":{"date-parts":[[2022,5,2]]}}}