{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T05:24:51Z","timestamp":1771478691871,"version":"3.50.1"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"23","license":[{"start":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T00:00:00Z","timestamp":1652313600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T00:00:00Z","timestamp":1652313600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100006360","name":"Bundesministerium f\u00fcr Wirtschaft und Energie","doi-asserted-by":"publisher","award":["FKZ 20X1905A-D"],"award-info":[{"award-number":["FKZ 20X1905A-D"]}],"id":[{"id":"10.13039\/501100006360","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003509","name":"\u00d6rebro University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003509","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>One-step reinforcement learning explanation methods account for individual actions but fail to consider the agent\u2019s future behavior, which can make their interpretation ambiguous. We propose to address this limitation by providing hierarchical goals as context for one-step explanations. By considering the current hierarchical goal as a context, one-step explanations can be interpreted with higher certainty, as the agent\u2019s future behavior is more predictable. We combine reward decomposition with hierarchical reinforcement learning into a novel explainable reinforcement learning framework, which yields more interpretable, goal-contextualized one-step explanations. With a qualitative analysis of one-step reward decomposition explanations, we first show that their interpretability is indeed limited in scenarios with multiple, different optimal policies\u2014a characteristic shared by other one-step explanation methods. Then, we show that our framework retains high interpretability in such cases, as the hierarchical goal can be considered as context for the explanation. To the best of our knowledge, our work is the first to investigate hierarchical goals not as an explanation directly but as additional context for one-step reinforcement learning explanations.<\/jats:p>","DOI":"10.1007\/s00521-022-07280-8","type":"journal-article","created":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T11:07:01Z","timestamp":1652353621000},"page":"16693-16704","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Hierarchical goals contextualize local reward decomposition explanations"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8151-4692","authenticated-orcid":false,"given":"Finn","family":"Rietz","sequence":"first","affiliation":[]},{"given":"Sven","family":"Magg","sequence":"additional","affiliation":[]},{"given":"Fredrik","family":"Heintz","sequence":"additional","affiliation":[]},{"given":"Todor","family":"Stoyanov","sequence":"additional","affiliation":[]},{"given":"Stefan","family":"Wermter","sequence":"additional","affiliation":[]},{"given":"Johannes A.","family":"Stork","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,5,12]]},"reference":[{"issue":"1\u20132","key":"7280_CR1","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/S0304-3800(02)00064-9","volume":"154","author":"JD Olden","year":"2002","unstructured":"Olden JD, Jackson DA (2002) Illuminating the \u201cblack box\u2019\u2019: a randomization approach for understanding variable contributions in artificial neural networks. Ecol Model 154(1\u20132):135\u2013150. https:\/\/doi.org\/10.1016\/S0304-3800(02)00064-9","journal-title":"Ecol Model"},{"issue":"6","key":"7280_CR2","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","volume":"34","author":"K Arulkumaran","year":"2017","unstructured":"Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep reinforcement learning: a brief survey. IEEE Signal Process Mag 34(6):26\u201338. https:\/\/doi.org\/10.1109\/MSP.2017.2743240","journal-title":"IEEE Signal Process Mag"},{"key":"7280_CR3","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106685","volume":"214","author":"A Heuillet","year":"2021","unstructured":"Heuillet A, Couthouis F, Rodr\u00edguez ND (2021) Explainability in deep reinforcement learning. Knowl Based Syst 214:106685. https:\/\/doi.org\/10.1016\/j.knosys.2020.106685","journal-title":"Knowl Based Syst"},{"key":"7280_CR4","unstructured":"Molnar C (2022) Interpretable machine learning, 2nd edn. https:\/\/christophm.github.io\/interpretable-ml-book\/cite.html"},{"key":"7280_CR5","doi-asserted-by":"publisher","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u201cWhy should I trust you?\u201d: explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135\u20131144. https:\/\/doi.org\/10.1145\/2939672.2939778","DOI":"10.1145\/2939672.2939778"},{"issue":"2","key":"7280_CR6","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","volume":"128","author":"RR Selvaraju","year":"2020","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2020) Grad-cam: visual explanations from deep networks via gradient-based localization. Int J Comput Vis 128(2):336\u2013359. https:\/\/doi.org\/10.1007\/s11263-019-01228-7","journal-title":"Int J Comput Vis"},{"key":"7280_CR7","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.550030","volume":"4","author":"L Wells","year":"2021","unstructured":"Wells L, Bednarz T (2021) Explainable AI and reinforcement learning - a systematic review of current approaches and trends. Front Artif Intell 4:550030. https:\/\/doi.org\/10.3389\/frai.2021.550030","journal-title":"Front Artif Intell"},{"key":"7280_CR8","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1007\/978-3-030-57321-8_5","volume-title":"Machine learning and knowledge extraction. Lecture notes in computer science","author":"E Puiutta","year":"2020","unstructured":"Puiutta E, Veith EMSP (2020) Explainable reinforcement learning: a survey. Machine learning and knowledge extraction. Lecture notes in computer science, vol 12279. Springer, Cham, pp 77\u201395"},{"key":"7280_CR9","doi-asserted-by":"publisher","unstructured":"Liu G, Schulte O, Zhu W, Li Q (2018) Toward interpretable deep reinforcement learning with linear model u-trees. In: Machine learning and knowledge discovery in databases - european conference, ECML PKDD 2018, Dublin, Ireland, September 10\u201314, 2018, proceedings, Part II. Lecture notes in computer science, vol. 11052, pp. 414\u2013429. https:\/\/doi.org\/10.1007\/978-3-030-10928-8_25","DOI":"10.1007\/978-3-030-10928-8_25"},{"issue":"5","key":"7280_CR10","doi-asserted-by":"publisher","first-page":"628","DOI":"10.1007\/s10458-019-09418-w","volume":"33","author":"O Amir","year":"2019","unstructured":"Amir O, Doshi-Velez F, Sarne D (2019) Summarizing agent strategies. Auton Agents Multi-Agent Syst 33(5):628\u2013644. https:\/\/doi.org\/10.1007\/s10458-019-09418-w","journal-title":"Auton Agents Multi-Agent Syst"},{"key":"7280_CR11","unstructured":"Amir D, Amir O (2018) Highlights: summarizing agent behavior to people. In: Proceedings of the 17th International conference on autonomous agents and multiagent systems. AAMAS 2018. International foundation for autonomous agents and multiagent systems, Richland, SC, pp. 1168\u20131176. http:\/\/dl.acm.org\/citation.cfm?id=3237869"},{"key":"7280_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2020.103367","author":"P Sequeira","year":"2020","unstructured":"Sequeira P, Gervasio MT (2020) Interestingness elements for explainable reinforcement learning: understanding agents\u2019 capabilities and limitations. Artif Intell. https:\/\/doi.org\/10.1016\/j.artint.2020.103367","journal-title":"Artif Intell"},{"key":"7280_CR13","doi-asserted-by":"publisher","unstructured":"Topin N, Veloso M (2019) Generation of policy-level explanations for reinforcement learning. In: The Thirty-Third AAAI conference on artificial intelligence, pp. 2514\u20132521. https:\/\/doi.org\/10.1609\/aaai.v33i01.33012514","DOI":"10.1609\/aaai.v33i01.33012514"},{"issue":"2","key":"7280_CR14","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1007\/s10514-018-9771-0","volume":"43","author":"SH Huang","year":"2019","unstructured":"Huang SH, Held D, Abbeel P, Dragan AD (2019) Enabling robots to communicate their objectives. Auton Robots 43(2):309\u2013326. https:\/\/doi.org\/10.1007\/s10514-018-9771-0","journal-title":"Auton Robots"},{"key":"7280_CR15","unstructured":"Zahavy T, Ben-Zrihem N, Mannor S (2016) Graying the black box: understanding dqns. In: Proceedings of the 33nd international conference on machine learning, ICML. JMLR Workshop and Conference Proceedings, vol. 48, pp. 1899\u20131908. http:\/\/proceedings.mlr.press\/v48\/zahavy16.html"},{"key":"7280_CR16","doi-asserted-by":"publisher","first-page":"171058","DOI":"10.1109\/ACCESS.2020.3023394","volume":"8","author":"A Alharin","year":"2020","unstructured":"Alharin A, Doan T, Sartipi M (2020) Reinforcement learning interpretation methods: a survey. IEEE Access 8:171058\u2013171077. https:\/\/doi.org\/10.1109\/ACCESS.2020.3023394","journal-title":"IEEE Access"},{"key":"7280_CR17","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103525","volume":"299","author":"R Dazeley","year":"2021","unstructured":"Dazeley R, Vamplew P, Foale C, Young C, Aryal S, Cruz F (2021) Levels of explainable artificial intelligence for human-aligned conversational explanations. Artif Intell 299:103525. https:\/\/doi.org\/10.1016\/j.artint.2021.103525","journal-title":"Artif Intell"},{"key":"7280_CR18","doi-asserted-by":"publisher","unstructured":"Alqaraawi A, Schuessler M, Wei\u00df P, Costanza E, Berthouze N (2020) Evaluating saliency map explanations for convolutional neural networks: a user study. In: IUI 2020: 25th international conference on intelligent user interfaces, Cagliari, Italy, March 17\u201320, 2020. ACM, New York, NY, USA, pp. 275\u2013285 https:\/\/doi.org\/10.1145\/3377325.3377519","DOI":"10.1145\/3377325.3377519"},{"key":"7280_CR19","doi-asserted-by":"publisher","unstructured":"Iyer R, Li Y, Li H, Lewis M, Sundar R, Sycara KP (2018) Transparency and explanation in deep reinforcement learning neural networks. In: Proceedings of the 2018 AAAI\/ACM Conference on AI, Ethics, and Society, AIES 2018, New Orleans, LA, USA, February 02-03, 2018, pp. 144\u2013150. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3278721.3278776","DOI":"10.1145\/3278721.3278776"},{"key":"7280_CR20","unstructured":"Atrey A, Clary K, Jensen DD (2020) Exploratory not explanatory: counterfactual analysis of saliency maps for deep reinforcement learning. In: 8th international conference on learning representations. https:\/\/openreview.net\/forum?id=rkl3m1BFDB"},{"issue":"1\u20132","key":"7280_CR21","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/S0004-3702(99)00052-1","volume":"112","author":"RS Sutton","year":"1999","unstructured":"Sutton RS, Precup D, Singh SP (1999) Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif Intell 112(1\u20132):181\u2013211. https:\/\/doi.org\/10.1016\/S0004-3702(99)00052-1","journal-title":"Artif Intell"},{"key":"7280_CR22","unstructured":"Kulkarni TD, Narasimhan K, Saeedi A, Tenenbaum J (2016) Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation. In: Advances in neural information processing systems, pp. 3675\u20133683. https:\/\/proceedings.neurips.cc\/paper\/2016\/hash\/f442d33fa06832082290ad8544a8da27-Abstract.html"},{"key":"7280_CR23","volume-title":"Reinforcement learning: an introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: an introduction, 2nd edn. The MIT Press, Cambridge, MA","edition":"2"},{"key":"7280_CR24","doi-asserted-by":"publisher","unstructured":"Beyret B, Shafti A, Faisal AA (2019) Dot-to-dot: explainable hierarchical reinforcement learning for robotic manipulation. In: 2019 IEEE\/RSJ international conference on intelligent robots and systems (IROS), pp. 5014\u20135019. https:\/\/doi.org\/10.1109\/IROS40897.2019.8968488","DOI":"10.1109\/IROS40897.2019.8968488"},{"key":"7280_CR25","unstructured":"Shu T, Xiong C, Socher R (2018) Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings, vol. 3, pp. 1\u201314"},{"key":"7280_CR26","unstructured":"Greydanus S, Koul A, Dodge J, Fern A (2018) Visualizing and understanding atari agents. In: Proceedings of the 35th international conference on machine learning. Proceedings of machine learning research, vol. 80, pp. 1787\u20131796. http:\/\/proceedings.mlr.press\/v80\/greydanus18a.html"},{"key":"7280_CR27","unstructured":"Yang Z, Bai S, Zhang L, Torr PHS (2018) Learn to interpret atari agents, arXiv preprint arXiv:1812.11276"},{"key":"7280_CR28","doi-asserted-by":"publisher","unstructured":"Annasamy RM, Sycara KP (2019) Towards better interpretability in deep q-networks. In: Proceedings of the AAAI conference on artificial intelligence, pp. 4561\u20134569. https:\/\/doi.org\/10.1609\/aaai.v33i01.33014561","DOI":"10.1609\/aaai.v33i01.33014561"},{"key":"7280_CR29","unstructured":"Mott A, Zoran D, Chrzanowski M, Wierstra D, Rezende DJ (2019) Towards interpretable reinforcement learning using attention augmented agents. In: Advances in neural information processing systems 32: annual conference on neural information processing systems 2019, pp. 12329\u201312338. https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/e9510081ac30ffa83f10b68cde1cac07-Abstract.html"},{"key":"7280_CR30","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103455","volume":"295","author":"ML Olson","year":"2021","unstructured":"Olson ML, Khanna R, Neal L, Li F, Wong W (2021) Counterfactual state explanations for reinforcement learning agents via generative deep learning. Artif Intell 295:103455. https:\/\/doi.org\/10.1016\/j.artint.2021.103455","journal-title":"Artif Intell"},{"key":"7280_CR31","first-page":"18375","volume":"33","author":"H Yau","year":"2020","unstructured":"Yau H, Russell C, Hadfield S (2020) What did you think would happen? Explaining agent behaviour through intended outcomes. Adv Neural Inf Process Syst 33:18375\u201318386","journal-title":"Adv Neural Inf Process Syst"},{"key":"7280_CR32","doi-asserted-by":"publisher","unstructured":"Dodson T, Mattei N, Goldsmith J (2011) A natural language argumentation interface for explanation generation in markov decision processes. In: Algorithmic Decision Theory. Lecture Notes in Computer Science, vol. 6992, pp. 42\u201355. https:\/\/doi.org\/10.1007\/978-3-642-24873-3_4","DOI":"10.1007\/978-3-642-24873-3_4"},{"key":"7280_CR33","unstructured":"van\u00a0der Waa J, van Diggelen J, van\u00a0den Bosch K, Neerincx MA (2018) Contrastive explanations for reinforcement learning in terms of expected consequences. In: IJCAI-18 Workshop on explainable AI arxiv:1807.08706"},{"key":"7280_CR34","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103571","volume":"301","author":"T Huber","year":"2021","unstructured":"Huber T, Weitz K, Andr\u00e9 E, Amir O (2021) Local and global explanations of agent behavior: integrating strategy summaries with saliency maps. Artif Intell 301:103571. https:\/\/doi.org\/10.1016\/j.artint.2021.103571","journal-title":"Artif Intell"},{"key":"7280_CR35","unstructured":"Russell SJ, Zimdars A (2003) Q-decomposition for reinforcement learning agents. In: Proceedings of the 20th international conference on machine learning (ICML-03), pp. 656\u2013663. http:\/\/www.aaai.org\/Library\/ICML\/2003\/icml03-086.php"},{"key":"7280_CR36","unstructured":"Juozapaitis Z, Koul A, Fern A, Erwig M, Doshi-Velez F (2019) Explainable reinforcement learning via reward decomposition. In: Proceedings of the IJCAI 2019 workshop on explainable artificial intelligence, pp. 47\u201353"},{"key":"7280_CR37","doi-asserted-by":"publisher","unstructured":"Anderson A, Dodge J, Sadarangani A, Juozapaitis Z, Newman E, Irvine J, Chattopadhyay S, Fern A, Burnett M (2019) Explaining reinforcement learning to mere mortals: an empirical study. In: Proceedings of the twenty-eighth international joint conference on artificial intelligence, IJCAI-19, pp 1328\u20131334. https:\/\/doi.org\/10.24963\/ijcai.2019\/184","DOI":"10.24963\/ijcai.2019\/184"},{"key":"7280_CR38","doi-asserted-by":"publisher","unstructured":"Sutton RS, Maei HR, Precup D, Bhatnagar S, Silver D, Szepesv\u00e1ri C, Wiewiora E (2009) Fast gradient-descent methods for temporal-difference learning with linear function approximation. In: Proceedings of the 26th annual international conference on machine learning, vol. 382. Association for Computing Machinery, New York, NY, USA, pp. 993\u20131000 https:\/\/doi.org\/10.1145\/1553374.1553501","DOI":"10.1145\/1553374.1553501"},{"key":"7280_CR39","unstructured":"Schulman J, Levine S, Abbeel P, Jordan MI, Moritz P (2015) Trust region policy optimization. In: Proceedings of the 32nd international conference on machine learning. JMLR Workshop and Conference Proceedings, vol. 37, pp. 1889\u20131897. http:\/\/proceedings.mlr.press\/v37\/schulman15.html"},{"key":"7280_CR40","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/BF00992698","volume":"8","author":"CJCH Watkins","year":"1992","unstructured":"Watkins CJCH, Dayan P (1992) Technical note q-learning. Mach Learn 8:279\u2013292. https:\/\/doi.org\/10.1007\/BF00992698","journal-title":"Mach Learn"},{"issue":"7540","key":"7280_CR41","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller MA, Fidjeland A, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529\u2013533. https:\/\/doi.org\/10.1038\/nature14236","journal-title":"Nature"},{"key":"7280_CR42","unstructured":"Schaul T, Quan J, Antonoglou I, Silver D (2016) Prioritized experience replay. In: 4th International conference on learning representations, ICLR. http:\/\/arxiv.org\/abs\/1511.05952"},{"key":"7280_CR43","doi-asserted-by":"crossref","unstructured":"van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: Proceedings of the Thirtieth AAAI conference on artificial intelligence, pp. 2094\u20132100. http:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI16\/paper\/view\/12389","DOI":"10.1609\/aaai.v30i1.10295"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07280-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-022-07280-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-07280-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T19:02:50Z","timestamp":1689188570000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-022-07280-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,12]]},"references-count":43,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["7280"],"URL":"https:\/\/doi.org\/10.1007\/s00521-022-07280-8","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,12]]},"assertion":[{"value":"2 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 April 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose and declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"All authors agreed with the content and gave explicit consent to submit and obtained consent from the responsible authorities at the organization where the work has been carried out.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}