{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T16:45:26Z","timestamp":1776185126159,"version":"3.50.1"},"reference-count":173,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,4,9]],"date-time":"2024-04-09T00:00:00Z","timestamp":1712620800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"name":"NSF","award":["IIS-2046640 (CAREER)"],"award-info":[{"award-number":["IIS-2046640 (CAREER)"]}]},{"DOI":"10.13039\/100019923","name":"U.S. Army Combat Capabilities Development Command Army Research Laboratory","doi-asserted-by":"crossref","award":["W911NF-13-2-0045 (ARL Cyber Security CRA)"],"award-info":[{"award-number":["W911NF-13-2-0045 (ARL Cyber Security CRA)"]}],"id":[{"id":"10.13039\/100019923","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of reinforcement learning (RL) agents in sequential decision-making settings. Equipped with this information, practitioners can better understand important questions about RL agents (especially those deployed in the real world), such as what the agents will do and why. Despite increased interest, there exists a gap in the literature for organizing the plethora of papers\u2014especially in a way that centers the sequential decision-making nature of the problem. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We propose three high-level categories: feature importance, learning process and Markov decision process, and policy-level. We overview techniques according to this taxonomy, highlighting challenges and opportunities for future work. We conclude by using these gaps to motivate and outline a roadmap for future work.<\/jats:p>","DOI":"10.1145\/3616864","type":"journal-article","created":{"date-parts":[[2023,8,26]],"date-time":"2023-08-26T10:35:35Z","timestamp":1693046135000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":156,"title":["Explainable Reinforcement Learning: A Survey and Comparative Review"],"prefix":"10.1145","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1150-4418","authenticated-orcid":false,"given":"Stephanie","family":"Milani","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5376-2801","authenticated-orcid":false,"given":"Nicholay","family":"Topin","sequence":"additional","affiliation":[{"name":"Inpleo Inc., Pittsburgh, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6738-238X","authenticated-orcid":false,"given":"Manuela","family":"Veloso","sequence":"additional","affiliation":[{"name":"J. P. Morgan AI Research, New York, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2256-8329","authenticated-orcid":false,"given":"Fei","family":"Fang","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,4,9]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015430"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376615"},{"key":"e_1_3_2_4_2","article-title":"A theory of abstraction in reinforcement learning","author":"Abel David","year":"2022","unstructured":"David Abel. 2022. A theory of abstraction in reinforcement learning. arXiv preprint arXiv:2203.00397 (2022).","journal-title":"arXiv preprint arXiv:2203.00397"},{"issue":"1","key":"e_1_3_2_5_2","first-page":"109","article-title":"Test, measurement, and evaluation: Understanding and use of the concepts in education","volume":"9","author":"Adom Dickson","year":"2020","unstructured":"Dickson Adom, Jephtar Adu Mensah, and Dennis Atsu Dake. 2020. Test, measurement, and evaluation: Understanding and use of the concepts in education. International Journal of Evaluation and Research in Education 9, 1 (2020), 109\u2013119.","journal-title":"International Journal of Evaluation and Research in Education"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543846"},{"key":"e_1_3_2_7_2","article-title":"On the importance of application-grounded experimental design for evaluating explainable ML methods","author":"Amarasinghe Kasun","year":"2022","unstructured":"Kasun Amarasinghe, Kit T. Rodolfa, S\u00e9rgio Jesus, Valerie Chen, Vladimir Balayan, Pedro Saleiro, Pedro Bizarro, Ameet Talwalkar, and Rayid Ghani. 2022. On the importance of application-grounded experimental design for evaluating explainable ML methods. arXiv preprint arXiv:2206.13503 (2022).","journal-title":"arXiv preprint arXiv:2206.13503"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02247-0_24"},{"key":"e_1_3_2_9_2","first-page":"1168","volume-title":"Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems","author":"Amir Dan","year":"2018","unstructured":"Dan Amir and Ofra Amir. 2018. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems. 1168\u20131176."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/3237383.3237877"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.5555\/3367032.3367221"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014561"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.1997.606886"},{"key":"e_1_3_2_15_2","volume-title":"Proceedings of the 8th International Conference on Learning Representations","author":"Atrey Akanksha","year":"2020","unstructured":"Akanksha Atrey, Kaleigh Clary, and David Jensen. 2020. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep RL. In Proceedings of the 8th International Conference on Learning Representations."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2798607"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022140919877"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-04083-2_11"},{"key":"e_1_3_2_19_2","first-page":"2494","volume-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS \u201918)","author":"Bastani Osbert","year":"2018","unstructured":"Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. 2018. Verifiable reinforcement learning via policy extraction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS \u201918). 2494\u20132504."},{"key":"e_1_3_2_20_2","article-title":"Relational inductive biases, deep learning, and graph networks","author":"Battaglia Peter W.","year":"2018","unstructured":"Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018).","journal-title":"arXiv preprint arXiv:1806.01261"},{"key":"e_1_3_2_21_2","article-title":"TripleTree: A versatile interpretable representation of black box agents and their environments","author":"Bewley Tom","year":"2020","unstructured":"Tom Bewley and Jonathan Lawry. 2020. TripleTree: A versatile interpretable representation of black box agents and their environments. CoRR abs\/2009.04743 (2020).","journal-title":"CoRR abs\/2009.04743"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8968488"},{"key":"e_1_3_2_23_2","volume-title":"Proceedings of the 9th International Conference on Learning Representations","author":"Bica Ioana","year":"2021","unstructured":"Ioana Bica, Daniel Jarrett, Alihan H\u00fcy\u00fck, and Mihaela van der Schaar. 2021. Learning \u201cwhat-if\u201d explanations for sequential decision-making. In Proceedings of the 9th International Conference on Learning Representations."},{"key":"e_1_3_2_24_2","article-title":"On the opportunities and risks of foundation models","author":"Bommasani Rishi","year":"2021","unstructured":"Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).","journal-title":"arXiv preprint arXiv:2108.07258"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.5555\/3086821"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"David A. Broniatowski. 2021. Psychological Foundations of Explainability and Interpretability in Artificial Intelligence. Technical Report NISTIR 8367. NIST.","DOI":"10.6028\/NIST.IR.8367"},{"key":"e_1_3_2_27_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christophe Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12228"},{"key":"e_1_3_2_29_2","unstructured":"Donald T. Campbell and Thomas D. Cook. 1979. Quasi-experimentation. Rand McNally Chicago IL."},{"key":"e_1_3_2_30_2","first-page":"2686","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Carroll Micah D.","year":"2022","unstructured":"Micah D. Carroll, Anca Dragan, Stuart Russell, and Dylan Hadfield-Menell. 2022. Estimating and penalizing induced preference shifts in recommender systems. In Proceedings of the International Conference on Machine Learning. 2686\u20132708."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1996.8.6.1135"},{"key":"e_1_3_2_32_2","article-title":"This looks like that: Deep learning for interpretable image recognition","author":"Chen Chaofan","year":"2019","unstructured":"Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su. 2019. This looks like that: Deep learning for interpretable image recognition. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS \u201919). 8930\u20138941.","journal-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS \u201919)."},{"key":"e_1_3_2_33_2","article-title":"Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning","author":"Chen Jianyu","year":"2022","unstructured":"Jianyu Chen, Shengbo Eben Li, and Masayoshi Tomizuka. 2022. Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems 23, 6 (2022), 5068\u20135078.","journal-title":"IEEE Transactions on Intelligent Transportation Systems"},{"key":"e_1_3_2_34_2","article-title":"Use-case-grounded simulations for explanation evaluation","author":"Chen Valerie","year":"2022","unstructured":"Valerie Chen, Nari Johnson, Nicholay Topin, Gregory Plumb, and Ameet Talwalkar. 2022. Use-case-grounded simulations for explanation evaluation. arXiv preprint arXiv:2206.02256 (2022).","journal-title":"arXiv preprint arXiv:2206.02256"},{"key":"e_1_3_2_35_2","first-page":"2048","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Cobbe Karl","year":"2020","unstructured":"Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. 2020. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the International Conference on Machine Learning. 2048\u20132056."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-35288-2_6"},{"key":"e_1_3_2_37_2","unstructured":"Francisco Cruz Richard Dazeley and Peter Vamplew. 2020. Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario. arXiv e-prints arXiv:2006.13615 (2020)."},{"key":"e_1_3_2_38_2","article-title":"Interpretable AI for policy-making in pandemics","author":"Custode Leonardo Lucio","year":"2022","unstructured":"Leonardo Lucio Custode and Giovanni Iacca. 2022. Interpretable AI for policy-making in pandemics. arXiv preprint arXiv:2204.04256 (2022).","journal-title":"arXiv preprint arXiv:2204.04256"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520304.3528897"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.epsr.2022.107932"},{"key":"e_1_3_2_41_2","first-page":"6936","article-title":"Interpreting a deep reinforcement learning model with conceptual embedding and performance analysis","author":"Dai Yinglong","year":"2022","unstructured":"Yinglong Dai, Haibin Ouyang, Hong Zheng, Han Long, and Xiaojun Duan. 2022. Interpreting a deep reinforcement learning model with conceptual embedding and performance analysis. Applied Intelligence 53, 6 (2022), 6936\u20136952.","journal-title":"Applied Intelligence"},{"key":"e_1_3_2_42_2","first-page":"2388","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Danesh Mohamad H.","year":"2021","unstructured":"Mohamad H. Danesh, Anurag Koul, Alan Fern, and Saeed Khorram. 2021. Re-understanding finite-state representations of recurrent policy networks. In Proceedings of the International Conference on Machine Learning. 2388\u20132397."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2018.00095"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.100.1.223"},{"key":"e_1_3_2_45_2","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).","journal-title":"arXiv preprint arXiv:1810.04805"},{"key":"e_1_3_2_46_2","article-title":"Toward interpretable-AI policies using evolutionary nonlinear decision trees for discrete-action systems","author":"Dhebar Yashesh","year":"2022","unstructured":"Yashesh Dhebar, Kalyanmoy Deb, Subramanya Nageshrao, Ling Zhu, and Dimitar Filev. 2022. Toward interpretable-AI policies using evolutionary nonlinear decision trees for discrete-action systems. IEEE Transactions on Cybernetics. Early access, June 23, 2022.","journal-title":"IEEE Transactions on Cybernetics."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-04083-2_10"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1002\/ail2.36"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.84.5.795"},{"key":"e_1_3_2_50_2","article-title":"Rationalization: A neural machine translation approach to generating natural language explanations","author":"Ehsan Upol","year":"2018","unstructured":"Upol Ehsan, Brent Harrison, Larry Chan, and Mark Riedl. 2018. Rationalization: A neural machine translation approach to generating natural language explanations. In Proceedings of the 1st AAAI\/ACM Conference on Artificial Intelligence, Ethics, and Society.","journal-title":"Proceedings of the 1st AAAI\/ACM Conference on Artificial Intelligence, Ethics, and Society."},{"key":"e_1_3_2_51_2","article-title":"Human-centered explainable AI: Towards a reflective sociotechnical approach","author":"Ehsan Upol","year":"2020","unstructured":"Upol Ehsan and Mark O. Riedl. 2020. Human-centered explainable AI: Towards a reflective sociotechnical approach. arXiv preprint arXiv:2002.01092 (2020).","journal-title":"arXiv preprint arXiv:2002.01092"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(71)90010-5"},{"key":"e_1_3_2_53_2","article-title":"Distilling a neural network into a soft decision tree","author":"Frosst Nicholas","year":"2017","unstructured":"Nicholas Frosst and Geoffrey Hinton. 2017. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09784 (2017).","journal-title":"arXiv preprint arXiv:1711.09784"},{"key":"e_1_3_2_54_2","article-title":"Explaining reinforcement learning policies through counterfactual trajectories","author":"Frost Julius","year":"2022","unstructured":"Julius Frost, Olivia Watkins, Eric Weiner, Pieter Abbeel, Trevor Darrell, Bryan Plummer, and Kate Saenko. 2022. Explaining reinforcement learning policies through counterfactual trajectories. arXiv preprint arXiv:2201.12462 (2022).","journal-title":"arXiv preprint arXiv:2201.12462"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0193743"},{"key":"e_1_3_2_56_2","first-page":"5683","volume-title":"Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS \u201918)","author":"Goel Vikash","year":"2018","unstructured":"Vikash Goel, Jameson Weng, and Pascal Poupart. 2018. Unsupervised video object segmentation for deep reinforcement learning. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS \u201918). 5683\u20135694."},{"key":"e_1_3_2_57_2","article-title":"Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions","author":"Gottesman Omer","year":"2020","unstructured":"Omer Gottesman, Joseph Futoma, Yao Liu, Sonali Parbhoo, Leo Anthony Celi, Emma Brunskill, and Finale Doshi-Velez. 2020. Interpretable off-policy evaluation in reinforcement learning by highlighting influential transitions. arXiv preprint arXiv:2002.03478 (2020).","journal-title":"arXiv preprint arXiv:2002.03478"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.90.1.1"},{"key":"e_1_3_2_59_2","first-page":"1792","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Greydanus Samuel","year":"2018","unstructured":"Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. 2018. Visualizing and understanding Atari agents. In Proceedings of the 35th International Conference on Machine Learning. 1792\u20131801."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v40i2.2850"},{"key":"e_1_3_2_61_2","article-title":"Explainable deep reinforcement learning for aircraft separation assurance","author":"Guo Wei","year":"2022","unstructured":"Wei Guo and Peng Wei. 2022. Explainable deep reinforcement learning for aircraft separation assurance. In Proceedings of the 4th Digital Avionics Systems Conference.","journal-title":"Proceedings of the 4th Digital Avionics Systems Conference."},{"key":"e_1_3_2_62_2","article-title":"EDGE: Explaining deep reinforcement learning policies","volume":"34","author":"Guo Wenbo","year":"2021","unstructured":"Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021. EDGE: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems 34 (2021), 1\u201315.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2019.02002"},{"key":"e_1_3_2_64_2","unstructured":"William H. Guss Cayden Codel Katja Hofmann Brandon Houghton Noburu Kuno Stephanie Milani Sharada Prasanna Mohanty Diego Perez Liebana Ruslan Salakhutdinov Nicholay Topin Manuela Veloso and Phillip Wang. 2019. The MineRL competition on sample efficient reinforcement learning using human priors. arXiv:1904.10079 (2019)."},{"key":"e_1_3_2_65_2","article-title":"Causes and explanations: A structural-model approach. Part II: Explanations","author":"Halpern Joseph Y.","year":"2005","unstructured":"Joseph Y. Halpern and Judea Pearl. 2005. Causes and explanations: A structural-model approach. Part II: Explanations. British Journal for the Philosophy of Science 56 (2005), 889\u2013911.","journal-title":"British Journal for the Philosophy of Science"},{"key":"e_1_3_2_66_2","article-title":"Bayesian persuasion for algorithmic recourse","author":"Harris Keegan","year":"2021","unstructured":"Keegan Harris, Valerie Chen, Joon Sik Kim, Ameet Talwalkar, Hoda Heidari, and Zhiwei Steven Wu. 2021. Bayesian persuasion for algorithmic recourse. arXiv preprint arXiv:2112.06283 (2021).","journal-title":"arXiv preprint arXiv:2112.06283"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i9.16935"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020233"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2017.07.005"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319619.3326755"},{"key":"e_1_3_2_71_2","article-title":"Deep reinforcement learning from self-play in imperfect-information games","author":"Heinrich Johannes","year":"2016","unstructured":"Johannes Heinrich and David Silver. 2016. Deep reinforcement learning from self-play in imperfect-information games. arXiv preprint arXiv:1603.01121 (2016).","journal-title":"arXiv preprint arXiv:1603.01121"},{"key":"e_1_3_2_72_2","unstructured":"Alexandre Heuillet Fabien Couthouis and Natalia D\u00edaz-Rodr\u00edguez. 2020. Explainability in deep reinforcement learning. arXiv:cs.AI\/2008.06693 (2020)."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1127647"},{"key":"e_1_3_2_74_2","article-title":"A benchmark for interpretability methods in deep neural networks","author":"Hooker Sara","year":"2019","unstructured":"Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS \u201919). 9737\u20139748.","journal-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems (NIPS \u201919)."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2018.8593649"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30179-8_16"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3054912"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364920987859"},{"key":"e_1_3_2_79_2","first-page":"6441","article-title":"Benchmarking deep learning interpretability in time series predictions","volume":"33","author":"Ismail Aya Abdelsalam","year":"2020","unstructured":"Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. 2020. Benchmarking deep learning interpretability in time series predictions. Advances in Neural Information Processing Systems 33 (2020), 6441\u20136452.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN52387.2021.9534363"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278776"},{"key":"e_1_3_2_82_2","article-title":"Lazy-MDPs: Towards interpretable reinforcement learning by learning when to act","author":"Jacq Alexis","year":"2022","unstructured":"Alexis Jacq, Johan Ferret, Olivier Pietquin, and Matthieu Geist. 2022. Lazy-MDPs: Towards interpretable reinforcement learning by learning when to act. arXiv preprint arXiv:2203.08542 (2022).","journal-title":"arXiv preprint arXiv:2203.08542"},{"key":"e_1_3_2_83_2","article-title":"Preprocessing reward functions for interpretability","author":"Jenner Erik","year":"2022","unstructured":"Erik Jenner and Adam Gleave. 2022. Preprocessing reward functions for interpretability. arXiv preprint arXiv:2203.13553 (2022).","journal-title":"arXiv preprint arXiv:2203.13553"},{"key":"e_1_3_2_84_2","article-title":"Policy Extraction via Online Q-Value Distillation","author":"Jhunjhunwala Aman","year":"2019","unstructured":"Aman Jhunjhunwala. 2019. Policy Extraction via Online Q-Value Distillation. Masters Thesis, University of Waterloo.","journal-title":"Masters Thesis, University of Waterloo."},{"key":"e_1_3_2_85_2","volume-title":"Proceedings of the 28th International Joint Conference on Artificial Intelligence Workshop on Explainable Artificial Intelligence","author":"Juozapaitis Zoe","year":"2019","unstructured":"Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. 2019. Explainable reinforcement learning via reward decomposition. In Proceedings of the 28th International Joint Conference on Artificial Intelligence Workshop on Explainable Artificial Intelligence."},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523111.3523127"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2021.3054625"},{"key":"e_1_3_2_88_2","first-page":"5637","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Koh Pang Wei","year":"2021","unstructured":"Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. 2021. WILDS: A benchmark of in-the-wild distribution shifts. In Proceedings of the International Conference on Machine Learning. 5637\u20135664."},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511816239"},{"key":"e_1_3_2_90_2","article-title":"Learning finite state representations of recurrent policy networks","author":"Koul Anurag","year":"2018","unstructured":"Anurag Koul, Sam Greydanus, and Alan Fern. 2018. Learning finite state representations of recurrent policy networks. arXiv preprint arXiv:1811.12530 (2018).","journal-title":"arXiv preprint arXiv:1811.12530"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.3102\/00028312021002435"},{"key":"e_1_3_2_92_2","article-title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","author":"Kulkarni Tejas D.","year":"2016","unstructured":"Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS \u201916). 1\u20139.","journal-title":"Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS \u201916)."},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.5555\/3367032.3367231"},{"key":"e_1_3_2_94_2","first-page":"5979","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Landajuela Mikel","year":"2021","unstructured":"Mikel Landajuela, Brenden K. Petersen, Sookyung Kim, Claudio P. Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F. Pettit, and Daniel Faissol. 2021. Discovering symbolic policies with deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. 5979\u20135989."},{"key":"e_1_3_2_95_2","article-title":"URLB: Unsupervised reinforcement learning benchmark","author":"Laskin Michael","year":"2021","unstructured":"Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. 2021. URLB: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:2110.15191 (2021).","journal-title":"arXiv preprint arXiv:2110.15191"},{"key":"e_1_3_2_96_2","article-title":"Data center cooling using model-predictive control","author":"Lazic Nevena","year":"2018","unstructured":"Nevena Lazic, Craig Boutilier, Tyler Lu, Eehern Wong, Binz Roy, M. K. Ryu, and Greg Imwalle. 2018. Data center cooling using model-predictive control. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS \u201918). 1\u201310.","journal-title":"Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS \u201918)."},{"key":"e_1_3_2_97_2","first-page":"3703","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Le Hoang","year":"2019","unstructured":"Hoang Le, Cameron Voloshin, and Yisong Yue. 2019. Batch policy learning under constraints. In Proceedings of the International Conference on Machine Learning. 3703\u20133712."},{"key":"e_1_3_2_98_2","article-title":"SoftGym: Benchmarking deep reinforcement learning for deformable object manipulation","author":"Lin Xingyu","year":"2020","unstructured":"Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. 2020. SoftGym: Benchmarking deep reinforcement learning for deformable object manipulation. arXiv preprint arXiv:2011.07215 (2020).","journal-title":"arXiv preprint arXiv:2011.07215"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3143518"},{"key":"e_1_3_2_100_2","article-title":"Tactics of adversarial attack on deep reinforcement learning agents","author":"Lin Yen-Chen","year":"2017","unstructured":"Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. 2017. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748 (2017).","journal-title":"arXiv preprint arXiv:1703.06748"},{"key":"e_1_3_2_101_2","article-title":"Contrastive explanations for reinforcement learning via embedded self predictions","author":"Lin Zhengxian","year":"2020","unstructured":"Zhengxian Lin, Kim-Ho Lam, and Alan Fern. 2020. Contrastive explanations for reinforcement learning via embedded self predictions. arXiv preprint arXiv:2010.05180 (2020).","journal-title":"arXiv preprint arXiv:2010.05180"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_2_103_2","first-page":"414","volume-title":"Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases","author":"Liu Guiliang","year":"2018","unstructured":"Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. 2018. Toward interpretable deep reinforcement learning with linear model U-trees. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 414\u2013429."},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i03.5631"},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33492-4_6"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.5555\/3535850.3535950"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1109\/IISA.2019.8900669"},{"key":"e_1_3_2_108_2","article-title":"MAVIPER: Learning decision tree policies for interpretable multi-agent reinforcement learning","author":"Milani Stephanie","year":"2022","unstructured":"Stephanie Milani, Zhicheng Zhang, Nicholay Topin, Zheyuan Ryan Shi, Charles Kamhoua, Evangelos E. Papalexakis, and Fei Fang. 2022. MAVIPER: Learning decision tree policies for interpretable multi-agent reinforcement learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases.","journal-title":"Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_3_2_110_2","article-title":"Playing Atari with deep reinforcement learning","author":"Mnih Volodymyr","year":"2013","unstructured":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).","journal-title":"arXiv preprint arXiv:1312.5602"},{"key":"e_1_3_2_111_2","article-title":"Model-based reinforcement learning: A survey","author":"Moerland Thomas M.","year":"2020","unstructured":"Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. 2020. Model-based reinforcement learning: A survey. arXiv preprint arXiv:2006.16712 (2020).","journal-title":"arXiv preprint arXiv:2006.16712"},{"key":"e_1_3_2_112_2","volume-title":"Interpretable Machine Learning","author":"Molnar Christoph","year":"2019","unstructured":"Christoph Molnar. 2019. Interpretable Machine Learning. Retrieved September 2, 2023 from https:\/\/christophm.github.io\/interpretable-ml-book\/"},{"key":"e_1_3_2_113_2","first-page":"12329","volume-title":"Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS \u201919)","author":"Mott Alexander","year":"2019","unstructured":"Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. 2019. Towards interpretable reinforcement learning using attention augmented agents. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS \u201919). 12329\u201312338."},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1002\/ejsp.2420150303"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3121870"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103455"},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1145\/235809.235811"},{"key":"e_1_3_2_118_2","doi-asserted-by":"crossref","unstructured":"Rohan Paleja Yaru Niu Andrew Silva Chace Ritchie Sugju Choi and Matthew Gombolay. 2022. Learning interpretable high-performing policies for autonomous driving. arXiv:2202.02352 (2022).","DOI":"10.15607\/RSS.2022.XVIII.068"},{"key":"e_1_3_2_119_2","article-title":"Align-RUDDER: Learning from few demonstrations by reward redistribution","author":"Patil Vihang P.","year":"2020","unstructured":"Vihang P. Patil, Markus Hofmarcher, Marius-Constantin Dinu, Matthias Dorfer, Patrick M. Blies, Johannes Brandstetter, Jose A. Arjona-Medina, and Sepp Hochreiter. 2020. Align-RUDDER: Learning from few demonstrations by reward redistribution. arXiv preprint arXiv:2009.14108 (2020).","journal-title":"arXiv preprint arXiv:2009.14108"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-15565-9_7"},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9116-6"},{"key":"e_1_3_2_122_2","article-title":"Explainable reinforcement learning: A survey","author":"Puiutta Erika","year":"2020","unstructured":"Erika Puiutta and Eric M. S. P. Veith. 2020. Explainable reinforcement learning: A survey. arXiv preprint arXiv:2005.06247 (2020).","journal-title":"arXiv preprint arXiv:2005.06247"},{"key":"e_1_3_2_123_2","article-title":"AI and the everything in the whole wide world benchmark","author":"Raji Inioluwa Deborah","year":"2021","unstructured":"Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. AI and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366 (2021).","journal-title":"arXiv preprint arXiv:2111.15366"},{"key":"e_1_3_2_124_2","article-title":"Hierarchical goals contextualize local reward decomposition explanations","author":"Rietz Finn","year":"2022","unstructured":"Finn Rietz, Sven Magg, Fredrik Heintz, Todor Stoyanov, Stefan Wermter, and Johannes A. Stork. 2022. Hierarchical goals contextualize local reward decomposition explanations. Neural Computing and Applications. Published online, May 12, 2022.","journal-title":"Neural Computing and Applications."},{"key":"e_1_3_2_125_2","article-title":"Optimization methods for interpretable differentiable decision trees in reinforcement learning","author":"Rodriguez Ivan Dario Jimenez","year":"2019","unstructured":"Ivan Dario Jimenez Rodriguez, Taylor W. Killian, Sung-Hyun Son, and Matthew C. Gombolay. 2019. Optimization methods for interpretable differentiable decision trees in reinforcement learning. arXiv preprint arXiv:1903.09338 (2019).","journal-title":"arXiv preprint arXiv:1903.09338"},{"key":"e_1_3_2_126_2","first-page":"627","volume-title":"Proceedings of the 14th International Conference on Artificial Intelligence and Statistics","author":"Ross St\u00e9phane","year":"2011","unstructured":"St\u00e9phane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. 627\u2013635."},{"key":"e_1_3_2_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2630823"},{"key":"e_1_3_2_128_2","doi-asserted-by":"publisher","DOI":"10.1214\/21-SS133"},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA164453"},{"key":"e_1_3_2_130_2","volume-title":"Proceedings of the 8th International Conference on Learning Representations","author":"Rupprecht Christian","year":"2020","unstructured":"Christian Rupprecht, Cyril Ibrahim, and Christopher J. Pal. 2020. Finding and visualizing weaknesses of deep reinforcement learning agents. In Proceedings of the 8th International Conference on Learning Representations."},{"key":"e_1_3_2_131_2","first-page":"262","volume-title":"Proceedings of the Conference on Robot Learning","author":"Rusu Andrei A.","year":"2017","unstructured":"Andrei A. Rusu, Matej Ve\u010der\u00edk, Thomas Roth\u00f6rl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. 2017. Sim-to-real robot learning from pixels with progressive nets. In Proceedings of the Conference on Robot Learning. 262\u2013270."},{"key":"e_1_3_2_132_2","article-title":"The MineRL BASALT competition on learning from human feedback","author":"Shah Rohin","year":"2021","unstructured":"Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, and Anca Dragan. 2021. The MineRL BASALT competition on learning from human feedback. arXiv preprint arXiv:2107.01969 (2021).","journal-title":"arXiv preprint arXiv:2107.01969"},{"key":"e_1_3_2_133_2","article-title":"M-Walk: Learning to walk over graphs using Monte Carlo tree search","author":"Shen Yelong","year":"2018","unstructured":"Yelong Shen, Jianshu Chen, Po-Sen Huang, Yuqing Guo, and Jianfeng Gao. 2018. M-Walk: Learning to walk over graphs using Monte Carlo tree search. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS \u201918). 1\u201312.","journal-title":"Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS \u201918)."},{"key":"e_1_3_2_134_2","article-title":"Self-supervised discovering of causal features: Towards interpretable reinforcement learning","author":"Shi Wenjie","year":"2020","unstructured":"Wenjie Shi, Zhuoyuan Wang, Shiji Song, and Gao Huang. 2020. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv preprint arXiv:2003.07069 (2020).","journal-title":"arXiv preprint arXiv:2003.07069"},{"key":"e_1_3_2_135_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"e_1_3_2_137_2","article-title":"An interpretable deep reinforcement learning approach to autonomous driving","author":"Song Zhihao","year":"2022","unstructured":"Zhihao Song, Yunpeng Jiang, Jianyi Zhang, Paul Weng, Dong Li, Wulong Liu, and Jianye Hao. 2022. An interpretable deep reinforcement learning approach to autonomous driving. In Proceedings of the IJCAI Workshop on Artificial Intelligence for Autonomous Driving.","journal-title":"Proceedings of the IJCAI Workshop on Artificial Intelligence for Autonomous Driving."},{"key":"e_1_3_2_138_2","doi-asserted-by":"crossref","unstructured":"Sarath Sreedharan Siddharth Srivastava and Subbarao Kambhampati. 2020. TLdR: Policy summarization for factored SSP problems using temporal abstractions. In Proceedings of the 30th International Conference on Automated Planning and Scheduling.","DOI":"10.1609\/icaps.v30i1.6671"},{"key":"e_1_3_2_139_2","doi-asserted-by":"publisher","DOI":"10.1109\/34.817409"},{"key":"e_1_3_2_140_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton Richard S.","year":"2018","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA."},{"key":"e_1_3_2_141_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-8126-4_6"},{"key":"e_1_3_2_142_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377930.3389847"},{"key":"e_1_3_2_143_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2020.101836"},{"key":"e_1_3_2_144_2","article-title":"A survey on Explainable Artificial Intelligence (XAI): Towards medical XAI","author":"Tjoa Erico","year":"2019","unstructured":"Erico Tjoa and Cuntai Guan. 2019. A survey on Explainable Artificial Intelligence (XAI): Towards medical XAI. arXiv preprint arXiv:1907.07374 (2019).","journal-title":"arXiv preprint arXiv:1907.07374"},{"key":"e_1_3_2_145_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.system.2008.08.002"},{"key":"e_1_3_2_146_2","article-title":"Iterative bounding MDPs: Learning interpretable policies via non-interpretable methods","author":"Topin Nicholay","year":"2021","unstructured":"Nicholay Topin, Stephanie Milani, Fei Fang, and Manuela Veloso. 2021. Iterative bounding MDPs: Learning interpretable policies via non-interpretable methods. arXiv preprint arXiv:2102.13045 (2021).","journal-title":"arXiv preprint arXiv:2102.13045"},{"key":"e_1_3_2_147_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012514"},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287566"},{"key":"e_1_3_2_149_2","article-title":"Hybrid reward architecture for reinforcement learning","author":"Van Seijen Harm","year":"2017","unstructured":"Harm Van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang. 2017. Hybrid reward architecture for reinforcement learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS \u201917). 5398\u20135408.","journal-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS \u201917)."},{"key":"e_1_3_2_150_2","unstructured":"Varun Ravi Varma. 2021. Interpretable Reinforcement Learning with the Regression Tsetlin Machine. Ph.D. Dissertation University of Gronigen."},{"key":"e_1_3_2_151_2","first-page":"5045","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Verma Abhinav","year":"2018","unstructured":"Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. 2018. Programmatically interpretable reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning. 5045\u20135054."},{"key":"e_1_3_2_152_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z"},{"key":"e_1_3_2_153_2","article-title":"Starcraft II: A new challenge for reinforcement learning","author":"Vinyals Oriol","year":"2017","unstructured":"Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich K\u00fcttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. 2017. Starcraft II: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017).","journal-title":"arXiv preprint arXiv:1708.04782"},{"key":"e_1_3_2_154_2","volume-title":"CauseOccam: Learning Interpretable Abstract Representations in Reinforcement Learning Environments via Model Sparsity","author":"Volodin Sergei","year":"2021","unstructured":"Sergei Volodin. 2021. CauseOccam: Learning Interpretable Abstract Representations in Reinforcement Learning Environments via Model Sparsity. Technical Report. Ecole Polytechnique Federale de Lausanne."},{"key":"e_1_3_2_155_2","first-page":"841","article-title":"Counterfactual explanations without opening the black box: Automated decisions and the GDPR","volume":"31","author":"Wachter Sandra","year":"2017","unstructured":"Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology 31 (2017), 841.","journal-title":"Harvard Journal of Law & Technology"},{"key":"e_1_3_2_156_2","first-page":"22457","volume-title":"Proceedings of the International Conference on Machine Learning","author":"W\u00e4ldchen Stephan","year":"2022","unstructured":"Stephan W\u00e4ldchen, Sebastian Pokutta, and Felix Huber. 2022. Training characteristic functions with reinforcement learning: XAI-methods play Connect Four. In Proceedings of the International Conference on Machine Learning. 22457\u201322474."},{"key":"e_1_3_2_157_2","doi-asserted-by":"publisher","DOI":"10.1109\/RO-MAN46459.2019.8956301"},{"key":"e_1_3_2_158_2","first-page":"151","volume-title":"Proceedings of the 30th Benelux Conference on Artificial Intelligence","author":"Weitkamp Laurens","year":"2018","unstructured":"Laurens Weitkamp, Elise van der Pol, and Zeynep Akata. 2018. Visual rationalizations in deep reinforcement learning for Atari games. In Proceedings of the 30th Benelux Conference on Artificial Intelligence. 151\u2013165."},{"key":"e_1_3_2_159_2","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.550030"},{"key":"e_1_3_2_160_2","doi-asserted-by":"publisher","DOI":"10.1093\/jigpal\/jzp049"},{"key":"e_1_3_2_161_2","article-title":"DayDreamer: World models for physical robot learning","author":"Wu Philipp","year":"2022","unstructured":"Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. 2022. DayDreamer: World models for physical robot learning. arXiv preprint arXiv:2206.14176 (2022).","journal-title":"arXiv preprint arXiv:2206.14176"},{"key":"e_1_3_2_162_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33015621"},{"key":"e_1_3_2_163_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00748"},{"key":"e_1_3_2_164_2","first-page":"18375","article-title":"What did you think would happen? Explaining agent behaviour through intended outcomes","volume":"33","author":"Yau Herman","year":"2020","unstructured":"Herman Yau, Chris Russell, and Simon Hadfield. 2020. What did you think would happen? Explaining agent behaviour through intended outcomes. Advances in Neural Information Processing Systems 33 (2020), 18375\u201318386.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_165_2","unstructured":"Chong-Ho Yu. 2010. Reliability of Self-Report Data. Retrieved September 2 2023 from https:\/\/www.creative-wisdom.com\/teaching\/WBI\/memory.shtml"},{"key":"e_1_3_2_166_2","first-page":"1094","volume-title":"Proceedings of the Conference on Robot Learning","author":"Yu Tianhe","year":"2020","unstructured":"Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. 2020. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Conference on Robot Learning. 1094\u20131100."},{"key":"e_1_3_2_167_2","doi-asserted-by":"publisher","DOI":"10.1109\/2.53"},{"key":"e_1_3_2_168_2","volume-title":"Proceedings of the 33rd International Conference on Machine Learning","author":"Zahavy Tom","year":"2016","unstructured":"Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding DQNs. In Proceedings of the 33rd International Conference on Machine Learning."},{"key":"e_1_3_2_169_2","doi-asserted-by":"crossref","unstructured":"Amber E. Zelvelder Marcus Westberg and Kary Fr\u00e4mling. 2021. Assessing explainability in reinforcement learning. In Explainable and Transparent AI and Multi-Agent Systems. Lecture Notes in Computer Science Vol. 12688. Springer 223\u2013240.","DOI":"10.1007\/978-3-030-82017-6_14"},{"key":"e_1_3_2_170_2","doi-asserted-by":"publisher","DOI":"10.17775\/CSEEJPES.2018.00520"},{"key":"e_1_3_2_171_2","first-page":"1","article-title":"Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis","author":"Zhang Hengzhe","year":"2020","unstructured":"Hengzhe Zhang, Aimin Zhou, and Xin Lin. 2020. Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis. Complex & Intelligent Systems 6 (2020), 1\u201313.","journal-title":"Complex & Intelligent Systems"},{"key":"e_1_3_2_172_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-86520-7_38"},{"key":"e_1_3_2_173_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3091885"},{"key":"e_1_3_2_174_2","first-page":"1","article-title":"\u201cWhy should you trust my explanation?\u201d","author":"Zhang Yujia","year":"2019","unstructured":"Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. 2019. \u201cWhy should you trust my explanation?\u201d Understanding uncertainty in LIME explanations. In Proceedings of the International Conference on Machine Learning AI for Social Good Workshop. 1\u20139.","journal-title":"Understanding uncertainty in LIME explanations."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3616864","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3616864","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3616864","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:49:13Z","timestamp":1750286953000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3616864"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,9]]},"references-count":173,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3616864"],"URL":"https:\/\/doi.org\/10.1145\/3616864","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,9]]},"assertion":[{"value":"2022-10-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}