{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T20:54:58Z","timestamp":1768424098961,"version":"3.49.0"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"26","license":[{"start":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T00:00:00Z","timestamp":1744416000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T00:00:00Z","timestamp":1744416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Reinforcement learning (RL) systems can be complex and non-interpretable, making it challenging for non-AI experts to understand or intervene in their decisions. This is due in part to the sequential nature of RL in which actions are chosen because of their likelihood of obtaining future rewards. However, RL agents discard the qualitative features of their training, making it difficult to recover user-understandable information for \u201cwhy\u201d an action is chosen. We propose a technique <jats:italic>Experiential Explanations<\/jats:italic> to generate counterfactual explanations by training <jats:italic>influence predictors<\/jats:italic> along with the RL policy. Influence predictors are models that learn how different sources of reward affect the agent in different states, thus restoring information about how the policy reflects the environment. Two human evaluation studies revealed that participants presented with Experiential Explanations were better able to correctly guess what an agent would do than those presented with other standard types of explanation. Participants also found that Experiential Explanations are more understandable, satisfying, complete, useful, and accurate. Qualitative analysis provides information on the factors of Experiential Explanations that are most useful and the desired characteristics that participants seek from the explanations.\n<\/jats:p>","DOI":"10.1007\/s00521-024-10951-3","type":"journal-article","created":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T07:25:16Z","timestamp":1744442716000},"page":"22255-22285","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Experiential Explanations for Reinforcement Learning"],"prefix":"10.1007","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1298-8568","authenticated-orcid":false,"given":"Amal","family":"Alabdulkarim","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Madhuri","family":"Singh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gennie","family":"Mansi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kaely","family":"Hall","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Upol","family":"Ehsan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark O.","family":"Riedl","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,4,12]]},"reference":[{"key":"10951_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","volume":"267","author":"T Miller","year":"2019","unstructured":"Miller T (2019) Explanation in artificial intelligence: insights from the social sciences. Artif Intell 267:1\u201338","journal-title":"Artif Intell"},{"key":"10951_CR2","doi-asserted-by":"crossref","unstructured":"Zelvelder AE, Westberg M, Fr\u00e4mling K (2021) Assessing explainability in reinforcement learning. In: International workshop on explainable, transparent autonomous agents and multi-agent systems, pp 223\u2013240. Springer","DOI":"10.1007\/978-3-030-82017-6_14"},{"key":"10951_CR3","unstructured":"Anjomshoae S, Najjar A, Calvaresi D, Fr\u00e4mling K (2019) Explainable agents and robots: results from a systematic literature review. In: Proceedings of the 18th international conference on autonomous agents and MultiAgent systems. AAMAS \u201919, pp. 1078\u20131088. International foundation for autonomous agents and MultiAgent systems, Richland, SC"},{"key":"10951_CR4","doi-asserted-by":"publisher","first-page":"171058","DOI":"10.1109\/ACCESS.2020.3023394","volume":"8","author":"A Alharin","year":"2020","unstructured":"Alharin A, Doan T-N, Sartipi M (2020) Reinforcement learning interpretation methods: a survey. IEEE Access 8:171058\u2013171077","journal-title":"IEEE Access"},{"key":"10951_CR5","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.550030","volume":"4","author":"L Wells","year":"2021","unstructured":"Wells L, Bednarz T (2021) Explainable AI and reinforcement learning-a systematic review of current approaches and trends. Front Artif Intell 4:550030","journal-title":"Front Artif Intell"},{"key":"10951_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106685","volume":"214","author":"A Heuillet","year":"2021","unstructured":"Heuillet A, Couthouis F, D\u00edaz-Rodr\u00edguez N (2021) Explainability in deep reinforcement learning. Knowl-Based Syst 214:106685. https:\/\/doi.org\/10.1016\/j.knosys.2020.106685. (Accessed 2022-09-30)","journal-title":"Knowl-Based Syst"},{"key":"10951_CR7","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1007\/978-3-030-57321-8_5","volume-title":"Machine learning and knowledge extraction. Lecture notes in computer science","author":"E Puiutta","year":"2020","unstructured":"Puiutta E, Veith EMSP (2020) Explainable reinforcement learning: a survey. In: Holzinger A, Kieseberg P, Tjoa AM, Weippl E (eds) Machine learning and knowledge extraction. Lecture notes in computer science. Springer, Cham, pp 77\u201395. https:\/\/doi.org\/10.1007\/978-3-030-57321-8_5"},{"key":"10951_CR8","doi-asserted-by":"publisher","DOI":"10.1145\/3616864","author":"S Milani","year":"2023","unstructured":"Milani S, Topin N, Veloso M, Fang F (2023) Explainable reinforcement learning: a survey and comparative review. ACM Comput Surv. https:\/\/doi.org\/10.1145\/3616864","journal-title":"ACM Comput Surv"},{"key":"10951_CR9","doi-asserted-by":"publisher","unstructured":"Chakraborti T, Sreedharan S, Kambhampati S (2020) The emerging landscape of explainable automated planning & decision making. In: Proceedings of the twenty-ninth international joint conference on artificial intelligence, pp. 4803\u20134811. International joint conferences on artificial intelligence organization, Yokohama, Japan. https:\/\/doi.org\/10.24963\/ijcai.2020\/669. https:\/\/www.ijcai.org\/proceedings\/2020\/669. Accessed 11 Mar 2022","DOI":"10.24963\/ijcai.2020\/669"},{"key":"10951_CR10","volume-title":"Reinforcement learning: an introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. A Bradford Book, Cambridge"},{"key":"10951_CR11","doi-asserted-by":"crossref","unstructured":"Chakraborti T, Sreedharan S, Kambhampati S (2021) The emerging landscape of explainable automated planning & decision making. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 4803\u20134811","DOI":"10.24963\/ijcai.2020\/669"},{"key":"10951_CR12","unstructured":"Juozapaitis Z, Koul A, Fern A, Erwig M, Doshi-Velez F (2019) Explainable reinforcement learning via reward decomposition. In: IJCAI\/ECAI workshop on explainable artificial intelligence"},{"key":"10951_CR13","doi-asserted-by":"crossref","unstructured":"Das D, Chernova S (2020) Leveraging rationales to improve human task performance. In: Proceedings of the 25th international conference on intelligent user interfaces, pp 510\u2013518","DOI":"10.1145\/3377325.3377512"},{"key":"10951_CR14","doi-asserted-by":"crossref","unstructured":"Anderson A, Dodge J, Sadarangani A, Juozapaitis Z, Newman E, Irvine J, Chattopadhyay S, Fern A, Burnett M (2019) Explaining reinforcement learning to mere mortals: an empirical study. CoRR arXiv:1903.09708","DOI":"10.24963\/ijcai.2019\/184"},{"key":"10951_CR15","doi-asserted-by":"crossref","unstructured":"Septon Y, Huber T, Andr\u00e9 E, Amir O (2023) Integrating policy summaries with reward decomposition for explaining reinforcement learning agents. In: International conference on practical applications of agents and multi-agent systems, pp 320\u2013332. Springer","DOI":"10.1007\/978-3-031-37616-0_27"},{"key":"10951_CR16","unstructured":"Waa J, Diggelen J, Bosch Kvd, Neerincx M (2018) Contrastive explanations for reinforcement learning in terms of expected consequences. arXiv:1807.08706"},{"key":"10951_CR17","doi-asserted-by":"crossref","unstructured":"Madumal P, Miller T, Sonenberg L, Vetere F (2020) Explainable reinforcement learning through a causal lens. In: Proceedings of the AAAI conference on artificial intelligence, vol. 34, pp 2493\u20132500","DOI":"10.1609\/aaai.v34i03.5631"},{"key":"10951_CR18","unstructured":"Sreedharan S, Soni U, Verma M, Srivastava S, Kambhampati S (2022) Bridging the gap: providing post-hoc symbolic explanations for sequential decision-making problems with inscrutable representations. In: International conference on learning representations. https:\/\/openreview.net\/forum?id=o-1v9hdSult"},{"key":"10951_CR19","unstructured":"Frost J, Watkins O, Weiner E, Abbeel P, Darrell T, Plummer B, Saenko K (2022) Explaining reinforcement learning policies through counterfactual trajectories. In: ICML workshop on human in the loop learning (HILL)"},{"key":"10951_CR20","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103455","volume":"295","author":"ML Olson","year":"2021","unstructured":"Olson ML, Khanna R, Neal L, Li F, Wong W-K (2021) Counterfactual state explanations for reinforcement learning agents via generative deep learning. Artif Intell 295:103455","journal-title":"Artif Intell"},{"key":"10951_CR21","unstructured":"Huber T, Demmler M, Mertes S, Olson ML, Andr\u00e9 E (2023) Ganterfactual-RL: understanding reinforcement learning agents\u2019 strategies through visual counterfactual explanations. In: Proceedings of the 2023 international conference on autonomous agents and multiagent systems, pp 1097\u20131106"},{"key":"10951_CR22","doi-asserted-by":"crossref","unstructured":"Ehsan U, Tambwekar P, Chan L, Harrison B, Riedl MO (2019) Automated rationale generation: a technique for explainable AI and its effects on human perceptions. In: Proceedings of the 24th international conference on intelligent user interfaces, pp 263\u2013274","DOI":"10.1145\/3301275.3302316"},{"key":"10951_CR23","unstructured":"Das D, Chernova S, Kim B (2024) State2explanation: concept-based explanations to benefit agent learning and user understanding. In: Proceedings of the 37th international conference on neural information processing systems. NIPS \u201923. Curran Associates Inc., Red Hook"},{"key":"10951_CR24","unstructured":"Ng AY, Harada D, Russell S (1999) Policy invariance under reward transformations: theory and application to reward shaping. In: Icml, vol. 99, pp 278\u2013287. Citeseer"},{"issue":"7540","key":"10951_CR25","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529\u2013533","journal-title":"Nature"},{"key":"10951_CR26","unstructured":"Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: International conference on machine learning, pp 1928\u20131937. PMLR"},{"key":"10951_CR27","unstructured":"Hafner D, Lillicrap T, Norouzi M, Ba J (2020) Mastering Atari with discrete world models. arXiv:2010.02193"},{"key":"10951_CR28","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103525","volume":"299","author":"R Dazeley","year":"2021","unstructured":"Dazeley R, Vamplew P, Foale C, Young C, Aryal S, Cruz F (2021) Levels of explainable artificial intelligence for human-aligned conversational explanations. Artif Intell 299:103525","journal-title":"Artif Intell"},{"key":"10951_CR29","unstructured":"Chevalier-Boisvert M, Willems L, Pal S (2018) Minimalistic Gridworld environment for OpenAI Gym. GitHub. https:\/\/github.com\/maximecb\/gym-minigrid"},{"key":"10951_CR30","unstructured":"Hafner D (2021) Benchmarking the spectrum of agent capabilities. arXiv:2109.06780"},{"issue":"5","key":"10951_CR31","doi-asserted-by":"publisher","first-page":"593","DOI":"10.3390\/electronics10050593","volume":"10","author":"J Zhou","year":"2021","unstructured":"Zhou J, Gandomi AH, Chen F, Holzinger A (2021) Evaluating the quality of machine learning explanations: a survey on methods and metrics. Electronics 10(5):593","journal-title":"Electronics"},{"key":"10951_CR32","doi-asserted-by":"publisher","first-page":"1096257","DOI":"10.3389\/fcomp.2023.1096257","volume":"5","author":"RR Hoffman","year":"2023","unstructured":"Hoffman RR, Mueller ST, Klein G, Litman J (2023) Measures for explainable AI: explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance. Front Comput Sci 5:1096257","journal-title":"Front Comput Sci"},{"key":"10951_CR33","unstructured":"Willems L (2018) RL Starter Files for MiniGrid. https:\/\/github.com\/lcswillems\/rl-starter-files\/tree\/4205e05b7905fec16519bc0802596673d86af018. Accessed 25 Sept 2022"},{"key":"10951_CR34","unstructured":"Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv:1707.06347"},{"issue":"2","key":"10951_CR35","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","volume":"3","author":"V Braun","year":"2006","unstructured":"Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77\u2013101","journal-title":"Qual Res Psychol"},{"key":"10951_CR36","unstructured":"Miles MB, Huberman AM, Salda\u00f1a J (2020) Qualitative data analysis: a methods sourcebook, 4th edn. SAGE, Los Angeles"},{"issue":"2","key":"10951_CR37","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/0266-352X(95)93866-H","volume":"17","author":"WT Chan","year":"1995","unstructured":"Chan WT, Chow YK, Liu LF (1995) Neural network: an alternative to pile driving formulas. Comput Geotech 17(2):135\u2013156. https:\/\/doi.org\/10.1016\/0266-352X(95)93866-H","journal-title":"Comput Geotech"},{"key":"10951_CR38","doi-asserted-by":"publisher","first-page":"1375490","DOI":"10.3389\/frobt.2024.1375490","volume":"11","author":"P Tambwekar","year":"2024","unstructured":"Tambwekar P, Gombolay M (2024) Towards reconciling usability and usefulness of policy explanations for sequential decision-making systems. Front Robot AI 11:1375490","journal-title":"Front Robot AI"},{"issue":"3\u20134","key":"10951_CR39","first-page":"1","volume":"11","author":"J Dodge","year":"2021","unstructured":"Dodge J, Khanna R, Irvine J, Lam K-H, Mai T, Lin Z, Kiddle N, Newman E, Anderson A, Raja S et al (2021) After-action review for AI (AAR\/AI). ACM Trans Interact Intell Syst (TiiS) 11(3\u20134):1\u201335","journal-title":"ACM Trans Interact Intell Syst (TiiS)"},{"key":"10951_CR40","unstructured":"Qian P, Unhelkar V (2022) Evaluating the role of interactivity on improving transparency in autonomous agents. In: Proceedings of the 21st international conference on autonomous agents and multiagent systems, pp 1083\u20131091"},{"issue":"268","key":"10951_CR41","first-page":"1","volume":"22","author":"A Raffin","year":"2021","unstructured":"Raffin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N (2021) Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res 22(268):1\u20138","journal-title":"J Mach Learn Res"},{"key":"10951_CR42","unstructured":"Stani\u0107 A, Tang Y, Ha D, Schmidhuber J (2022) Learning to generalize with object-centric agents in the open world survival game crafter"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-024-10951-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-024-10951-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-024-10951-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,6]],"date-time":"2025-09-06T11:00:19Z","timestamp":1757156419000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-024-10951-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,12]]},"references-count":42,"journal-issue":{"issue":"26","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["10951"],"URL":"https:\/\/doi.org\/10.1007\/s00521-024-10951-3","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,12]]},"assertion":[{"value":"1 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 December 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 April 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflict of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}