{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T06:41:31Z","timestamp":1772865691185,"version":"3.50.1"},"reference-count":105,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,12,19]],"date-time":"2023-12-19T00:00:00Z","timestamp":1702944000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>\n            Sample inefficiency of deep reinforcement learning methods is a major obstacle for their use in real-world tasks as they naturally feature sparse rewards. In fact, this from-scratch approach is often impractical in environments where extreme negative outcomes are possible. Recent advances in imitation learning have improved sample efficiency by leveraging expert demonstrations. Most work along this line of research employs neural network-based approaches to recover an expert cost function. However, the complexity and lack of transparency make neural networks difficult to trust and deploy in the real world. In contrast, we present a method for extracting\n            <jats:italic>interpretable<\/jats:italic>\n            symbolic reward functions from expert data, which offers several advantages. First, the learned reward function can be parsed by a human to understand, verify and predict the behavior of the agent. Second, the reward function can be improved and modified by an expert. Finally, the structure of the reward function can be leveraged to extract\n            <jats:italic>explanations<\/jats:italic>\n            that encode richer domain knowledge than standard scalar rewards. To this end, we use an autoregressive recurrent neural network that generates hierarchical symbolic rewards represented by simple symbolic trees. The recurrent neural network is trained via risk-seeking policy gradients. We test our method in MuJoCo environments as well as a chemical plant simulator. We show that the discovered rewards can significantly accelerate the training process and achieve similar or better performance than neural network-based algorithms.\n          <\/jats:p>","DOI":"10.1145\/3627822","type":"journal-article","created":{"date-parts":[[2023,10,13]],"date-time":"2023-10-13T15:27:13Z","timestamp":1697210833000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Interpretable Imitation Learning with Symbolic Rewards"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9856-0038","authenticated-orcid":false,"given":"Nicolas","family":"Bougie","sequence":"first","affiliation":[{"name":"NEC-AIST AI Cooperative Research Laboratory, National Institute of Advanced Industrial Science and Technology, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1018-7613","authenticated-orcid":false,"given":"Takashi","family":"Onishi","sequence":"additional","affiliation":[{"name":"NEC-AIST AI Cooperative Research Laboratory, National Institute of Advanced Industrial Science and Technology, Japan and NEC Corporation Data Science Research Laboratories, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0707-1077","authenticated-orcid":false,"given":"Yoshimasa","family":"Tsuruoka","sequence":"additional","affiliation":[{"name":"NEC-AIST AI Cooperative Research Laboratory, National Institute of Advanced Industrial Science and Technology, Japan and Department of Information and Communication Engineering, The University of Tokyo, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,12,19]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Sanity checks for saliency maps","volume":"31","author":"Adebayo Julius","year":"2018","unstructured":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. Adv. Neural Inf. Process. Syst. 31 (2018), 9525\u20139536.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_3_2","article-title":"Demystifying black-box models with symbolic metamodels","volume":"32","author":"Alaa Ahmed M.","year":"2019","unstructured":"Ahmed M. Alaa and Mihaela van der Schaar. 2019. Demystifying black-box models with symbolic metamodels. Adv. Neural Inf. Process. Syst. 32 (2019), 11304\u201311314.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_4_2","article-title":"Verifiable reinforcement learning via policy extraction","volume":"31","author":"Bastani Osbert","year":"2018","unstructured":"Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. 2018. Verifiable reinforcement learning via policy extraction. Adv. Neural Inf. Process. Syst. 31 (2018), 2499\u20132509.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_5_2","first-page":"459","volume-title":"International Conference on Machine Learning","author":"Bello Irwan","year":"2017","unstructured":"Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. 2017. Neural optimizer search with reinforcement learning. In International Conference on Machine Learning. PMLR, 459\u2013468."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i13.17360"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8968488"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-61616-8_22"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05845-8"},{"key":"e_1_3_2_10_2","unstructured":"Yuri Burda Harrison Edwards Amos Storkey and Oleg Klimov. 2019. Exploration by random network distillation. In International Conference on Learning Representations . Retrieved from https:\/\/openreview.net\/forum?id=H1lJJnR5Ym"},{"key":"e_1_3_2_11_2","unstructured":"Kyunghyun Cho Bart van Merri\u00ebnboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP\u201914) 1724\u20131734."},{"key":"e_1_3_2_12_2","unstructured":"Paul F. Christiano Jan Leike Tom Brown Miljan Martic Shane Legg and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems 30 (2017)."},{"key":"e_1_3_2_13_2","first-page":"1","volume-title":"Proceedings of the IJCAI Workshop on Explainable Artificial Intelligence","author":"Coppens Youri","year":"2019","unstructured":"Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Now\u00e9, Tim Miller, Rosina Weber, and Daniele Magazzeni. 2019. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI Workshop on Explainable Artificial Intelligence. 1\u20136."},{"key":"e_1_3_2_14_2","first-page":"225","volume-title":"Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems","author":"Devlin Sam","year":"2011","unstructured":"Sam Devlin and Daniel Kudenko. 2011. Theoretical considerations of potential-based reward shaping for multi-agent systems. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems. ACM, 225\u2013232."},{"key":"e_1_3_2_15_2","article-title":"Explanations based on the missing: Towards contrastive explanations with pertinent negatives","volume":"31","author":"Dhurandhar Amit","year":"2018","unstructured":"Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Adv. Neural Inf. Process. Syst. 31 (2018).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(94)90047-7"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.23919\/MIPRO.2018.8400040"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359786"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007694015589"},{"key":"e_1_3_2_20_2","unstructured":"Dumitru Erhan Yoshua Bengio Aaron Courville and Pascal Vincent. 2009. Visualizing higher-layer features of a deep network. University of Montreal 1341 3 (2009) 1."},{"key":"e_1_3_2_21_2","first-page":"49","volume-title":"International Conference on Machine Learning","author":"Finn Chelsea","year":"2016","unstructured":"Chelsea Finn, Sergey Levine, and Pieter Abbeel. 2016. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning. PMLR, 49\u201358."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11794"},{"key":"e_1_3_2_23_2","article-title":"Generalizing across multi-objective reward functions in deep reinforcement learning","author":"Friedman Eli","year":"2018","unstructured":"Eli Friedman and Fred Fontaine. 2018. Generalizing across multi-objective reward functions in deep reinforcement learning. arXiv:1809.06364. Retrieved from https:\/\/arxiv.org\/abs\/1809.06364","journal-title":"arXiv:1809.06364"},{"key":"e_1_3_2_24_2","article-title":"Towards symbolic reinforcement learning with common sense","author":"Garcez Artur d\u2019Avila","year":"2018","unstructured":"Artur d\u2019Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso. 2018. Towards symbolic reinforcement learning with common sense. arXiv:1804.08597. Retrieved from https:\/\/arxiv.org\/abs\/1804.08597","journal-title":"arXiv:1804.08597"},{"key":"e_1_3_2_25_2","article-title":"Towards deep symbolic reinforcement learning","author":"Garnelo Marta","year":"2016","unstructured":"Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. 2016. Towards deep symbolic reinforcement learning. arXiv:1609.05518. Retrieved from https:\/\/arxiv.org\/abs\/1906.05518","journal-title":"arXiv:1609.05518"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_2_27_2","unstructured":"Claire Glanois Paul Weng Matthieu Zimmer Dong Li Tianpei Yang Jianye Hao and Wulong Liu. 2021. A survey on interpretable reinforcement learning. arXiv preprint arXiv:2112.13112 (2021)."},{"key":"e_1_3_2_28_2","first-page":"2376","volume-title":"International Conference on Machine Learning","author":"Goyal Yash","year":"2019","unstructured":"Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual visual explanations. In International Conference on Machine Learning. PMLR, 2376\u20132384."},{"key":"e_1_3_2_29_2","article-title":"Variational intrinsic control","author":"Gregor Karol","year":"2016","unstructured":"Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. 2016. Variational intrinsic control. arXiv:1611.07507. Retrieved from https:\/\/arxiv.org\/abs\/1611.07507","journal-title":"arXiv:1611.07507"},{"key":"e_1_3_2_30_2","first-page":"17","volume-title":"Proceedings AAMAS Workshop on Adaptive and Learning Agents and Multi-Agent Systems (ALAMAS-ALAg \u201908)","author":"Grzes Marek","year":"2008","unstructured":"Marek Grzes and Daniel Kudenko. 2008. Learning potential for reward shaping in reinforcement learning with tile coding. In Proceedings AAMAS Workshop on Adaptive and Learning Agents and Multi-Agent Systems (ALAMAS-ALAg \u201908). 17\u201323."},{"key":"e_1_3_2_31_2","first-page":"12222","article-title":"Edge: Explaining deep reinforcement learning policies","volume":"34","author":"Guo Wenbo","year":"2021","unstructured":"Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. 2021. Edge: Explaining deep reinforcement learning policies. Adv. Neural Inf. Process. Syst. 34 (2021), 12222\u201312236.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_32_2","first-page":"303","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction","author":"Hayes Bradley","year":"2017","unstructured":"Bradley Hayes and Julie A. Shah. 2017. Improving robot controller transparency through autonomous policy explanation. In Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction. IEEE, 303\u2013312."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ast.2021.107052"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2018.09.007"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11757"},{"key":"e_1_3_2_36_2","first-page":"4565","volume-title":"Proceedings of Advances in Neural Information Processing Systems","author":"Ho Jonathan","year":"2016","unstructured":"Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. In Proceedings of Advances in Neural Information Processing Systems. 4565\u20134573."},{"key":"e_1_3_2_37_2","volume-title":"International Conference on Learning Representations","author":"H\u00fcy\u00fck Alihan","year":"2021","unstructured":"Alihan H\u00fcy\u00fck, Daniel Jarrett, Cem Tekin, and Mihaela Van Der Schaar. 2021. Explaining by imitating: Understanding decisions by interpretable policy learning. In International Conference on Learning Representations."},{"key":"e_1_3_2_38_2","volume-title":"IJCAI\/ECAI Workshop on Explainable Artificial Intelligence","author":"Juozapaitis Zoe","year":"2019","unstructured":"Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. 2019. Explainable reinforcement learning via reward decomposition. In IJCAI\/ECAI Workshop on Explainable Artificial Intelligence."},{"key":"e_1_3_2_39_2","unstructured":"Pierre-Alexandre Kamienny St\u00e9phane d\u2019Ascoli Guillaume Lample and Fran\u00e7ois Charton. 2022. End-to-end symbolic regression with transformers. Advances in Neural Information Processing Systems 35 (2022) 10269\u201310281."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00169"},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/978-3-030-28954-6_14","volume-title":"Explainable AI: Interpreting, Explaining and Visualizing Deep Learning","author":"Kindermans Pieter-Jan","year":"2019","unstructured":"Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Sch\u00fctt, Sven D\u00e4hne, Dumitru Erhan, and Been Kim. 2019. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 267\u2013280."},{"key":"e_1_3_2_42_2","article-title":"Adam: A method for stochastic optimization","author":"Kingma Diederik P.","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980. Retrieved from https:\/\/arxiv.org\/abs\/1412.6980","journal-title":"arXiv:1412.6980"},{"key":"e_1_3_2_43_2","first-page":"5979","volume-title":"International Conference on Machine Learning","author":"Landajuela Mikel","year":"2021","unstructured":"Mikel Landajuela, Brenden K. Petersen, Sookyung Kim, Claudio P. Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F. Pettit, and Daniel Faissol. 2021. Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning. PMLR, 5979\u20135989."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-08987-4"},{"key":"e_1_3_2_45_2","volume-title":"Theory and Application of Reward Shaping in Reinforcement Learning","author":"Laud Adam Daniel","year":"2004","unstructured":"Adam Daniel Laud. 2004. Theory and Application of Reward Shaping in Reinforcement Learning. Ph. D. Dissertation. Advisor(s) Dejong, Gerald. AAI3130966."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2016.07.004"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2020.103568"},{"key":"e_1_3_2_48_2","unstructured":"Zhengxian Lin Kin-Ho Lam and Alan Fern. 2021. Contrastive explanations for reinforcement learning via embedded self predictions. In International Conference on Learning Representations . Retrieved from https:\/\/openreview.net\/forum?id=Ud3DSz72nYR"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.3390\/e23010018"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1358246100005130"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-10928-8_25"},{"key":"e_1_3_2_52_2","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Lundberg Scott M.","year":"2017","unstructured":"Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_53_2","unstructured":"Ronny Luss Amit Dhurandhar and Miao Liu. 2022. Interpreting reinforcement policies through local behaviors. Retrieved from https:\/\/openreview.net\/forum?id=7qaCQiuOVf"},{"issue":"7","key":"e_1_3_2_54_2","first-page":"533","article-title":"Vinyl Acetate Monomer (VAM) plant model: A new benchmark problem for control and operation study","volume":"49","author":"Machida Yuta","year":"2016","unstructured":"Yuta Machida, Shigeki Ootakara, Hiroya Seki, Yoshihiro Hashimoto, Manabu Kano, Yasuhiro Miyake, Naoto Anzai, Masayoshi Sawai, Takashi Katsuno, and Toshiaki Omata. 2016. Vinyl Acetate Monomer (VAM) plant model: A new benchmark problem for control and operation study. Proc. IFAC Symp. Dynam. Contr. Process Syst. Incl. Biosyst. 49, 7 (2016), 533\u2013538.","journal-title":"Proc. IFAC Symp. Dynam. Contr. Process Syst. Incl. Biosyst."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11741"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273572"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-26412-2_16"},{"key":"e_1_3_2_58_2","first-page":"29529","article-title":"Ella: Exploration through learned language abstraction","volume":"34","author":"Mirchandani Suvir","year":"2021","unstructured":"Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. 2021. Ella: Exploration through learned language abstraction. Adv. Neural Inf. Process. Syst. 34 (2021), 29529\u201329540.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/3927.001.0001"},{"key":"e_1_3_2_60_2","first-page":"1928","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Mnih Volodymyr","year":"2016","unstructured":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning. 1928\u20131937."},{"key":"e_1_3_2_61_2","volume-title":"Interpretable Machine Learning","author":"Molnar Christoph","year":"2020","unstructured":"Christoph Molnar. 2020. Interpretable Machine Learning. Lulu.com."},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dsp.2017.10.011"},{"key":"e_1_3_2_63_2","article-title":"Towards interpretable reinforcement learning using attention augmented agents","volume":"32","author":"Mott Alexander","year":"2019","unstructured":"Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. 2019. Towards interpretable reinforcement learning using attention augmented agents. Adv. Neural Inf. Process. Syst. 32 (2019).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8463162"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403186"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.5555\/3305890.3305968"},{"key":"e_1_3_2_67_2","first-page":"4095","volume-title":"International Conference on Machine Learning","author":"Pham Hieu","year":"2018","unstructured":"Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning. PMLR, 4095\u20134104."},{"key":"e_1_3_2_68_2","article-title":"Model agnostic supervised local explanations","volume":"31","author":"Plumb Gregory","year":"2018","unstructured":"Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. 2018. Model agnostic supervised local explanations. Advances in Neural Information Processing Systems 31 (2018).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-57321-8_5"},{"key":"e_1_3_2_70_2","first-page":"463","volume-title":"International Conference on Machine Learning (ICML \u201998)","volume":"98","author":"Randl\u00f8v Jette","year":"1998","unstructured":"Jette Randl\u00f8v and Preben Alstr\u00f8m. 1998. Learning to drive a bicycle using reinforcement learning and shaping. In International Conference on Machine Learning (ICML \u201998), Vol. 98. 463\u2013471."},{"key":"e_1_3_2_71_2","doi-asserted-by":"crossref","unstructured":"Marco Tulio Ribeiro Sameer Singh and Carlos Guestrin. 2016. \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201916) . Association for Computing Machinery 1135\u20131144.","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0048-x"},{"key":"e_1_3_2_73_2","first-page":"4442","volume-title":"International Conference on Machine Learning","author":"Sahoo Subham","year":"2018","unstructured":"Subham Sahoo, Christoph Lampert, and Georg Martius. 2018. Learning equations for extrapolation and control. In International Conference on Machine Learning. PMLR, 4442\u20134450."},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-53734-9_4"},{"key":"e_1_3_2_75_2","first-page":"1889","volume-title":"International Conference on Machine Learning","author":"Schulman John","year":"2015","unstructured":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning. PMLR, 1889\u20131897."},{"key":"e_1_3_2_76_2","article-title":"Proximal policy optimization algorithms","author":"Schulman John","year":"2017","unstructured":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. Retrieved from https:\/\/arxiv.org\/abs\/1707.06347","journal-title":"arXiv:1707.06347"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2020.103367"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN55064.2022.9892256"},{"key":"e_1_3_2_79_2","unstructured":"Tianmin Shu Caiming Xiong and Richard Socher. 2018. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In International Conference on Learning Representations . Retrieved from https:\/\/openreview.net\/forum?id=SJJQVZW0b"},{"key":"e_1_3_2_80_2","article-title":"Neural-encoding human experts\u2019 domain knowledge to warm start reinforcement learning","author":"Silva Andrew","year":"2019","unstructured":"Andrew Silva and Matthew Gombolay. 2019. Neural-encoding human experts\u2019 domain knowledge to warm start reinforcement learning. arXiv:1902.06007. Retrieved from https:\/\/arxiv.org\/abs\/1902.06007","journal-title":"arXiv:1902.06007"},{"key":"e_1_3_2_81_2","unstructured":"Sarath Sreedharan Utkarsh Soni Mudit Verma Siddharth Srivastava and Subbarao Kambhampati. 2022. Bridging the Gap: Providing post-hoc symbolic explanations for sequential decision-making problems with inscrutable representations. In The 10th International Conference on Learning Representations (ICLR\u201922) . OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=o-1v9hdSult"},{"key":"e_1_3_2_82_2","article-title":"A refinement-based architecture for knowledge representation and reasoning in robotics","author":"Sridharan Mohan","year":"2015","unstructured":"Mohan Sridharan, Michael Gelfond, Shiqi Zhang, and Jeremy Wyatt. 2015. A refinement-based architecture for knowledge representation and reasoning in robotics. arXiv:1508.03891. Retrieved from https:\/\/arxiv.org\/abs\/1508.03891","journal-title":"arXiv:1508.03891"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcss.2007.08.009"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.5555\/3378680.3378879"},{"key":"e_1_3_2_85_2","first-page":"414","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Tang Yujin","year":"2020","unstructured":"Yujin Tang, Duong Nguyen, and David Ha. 2020. Neuroevolution of self-interpretable agents. In Proceedings of the Genetic and Evolutionary Computation Conference. 414\u2013424."},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386109"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012514"},{"issue":"11","key":"e_1_3_2_88_2","article-title":"Visualizing data using t-SNE.","volume":"9","author":"Maaten Laurens Van der","year":"2008","unstructured":"Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 11 (2008).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_89_2","article-title":"Contrastive explanations for reinforcement learning in terms of expected consequences","author":"Waa Jasper van der","year":"2018","unstructured":"Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. 2018. Contrastive explanations for reinforcement learning in terms of expected consequences. arXiv:1807.08706. Retrieved from https:\/\/arxiv.org\/abs\/1807.08706","journal-title":"arXiv:1807.08706"},{"key":"e_1_3_2_90_2","first-page":"5045","volume-title":"International Conference on Machine Learning","author":"Verma Abhinav","year":"2018","unstructured":"Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. 2018. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning. PMLR, 5045\u20135054."},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3449726.3463166"},{"key":"e_1_3_2_92_2","first-page":"841","article-title":"Counterfactual explanations without opening the black box: Automated decisions and the GDPR","volume":"31","author":"Wachter Sandra","year":"2017","unstructured":"Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harv. JL Tech. 31 (2017), 841.","journal-title":"Harv. JL Tech."},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6220"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992696"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-020-09451-0"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11498"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01769885"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2022.3204236"},{"key":"e_1_3_2_99_2","first-page":"1899","volume-title":"International Conference on Machine Learning","author":"Zahavy Tom","year":"2016","unstructured":"Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. 2016. Graying the black box: Understanding dqns. In International Conference on Machine Learning. PMLR, 1899\u20131908."},{"key":"e_1_3_2_100_2","volume-title":"International Conference on Learning Representations","author":"Zambaldi Vinicius","year":"2019","unstructured":"Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et\u00a0al. 2019. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations."},{"key":"e_1_3_2_101_2","article-title":"Faster and safer training by embedding high-level knowledge into deep reinforcement learning","author":"Zhang Haodi","year":"2019","unstructured":"Haodi Zhang, Zihang Gao, Yi Zhou, Hao Zhang, Kaishun Wu, and Fangzhen Lin. 2019. Faster and safer training by embedding high-level knowledge into deep reinforcement learning. arXiv:1910.09986. Retrieved from https:\/\/arxiv.org\/abs\/1910.09986","journal-title":"arXiv:1910.09986"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/TETCI.2021.3100641"},{"key":"e_1_3_2_103_2","first-page":"1433","volume-title":"Proceedings of the AAAI International Conference on Artificial Intelligence (AAAI \u201908)","volume":"8","author":"Ziebart Brian D.","year":"2008","unstructured":"Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, Anind K. Dey, et\u00a0al. 2008. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI International Conference on Artificial Intelligence (AAAI \u201908), Vol. 8. 1433\u20131438."},{"key":"e_1_3_2_104_2","article-title":"Fine-tuning language models from human preferences","author":"Ziegler Daniel M.","year":"2019","unstructured":"Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv:1909.08593. Retrieved from https:\/\/arxiv.org\/abs\/1909.08593","journal-title":"arXiv:1909.08593"},{"key":"e_1_3_2_105_2","unstructured":"Barret Zoph and Quoc Le. 2017. Neural architecture search with reinforcement learning. In International Conference on Learning Representations . Retrieved from https:\/\/openreview.net\/forum?id=r1Ue8Hcxg"},{"key":"e_1_3_2_106_2","article-title":"Reward shaping via meta-learning","author":"Zou Haosheng","year":"2019","unstructured":"Haosheng Zou, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. 2019. Reward shaping via meta-learning. arXiv:1901.09330. Retrieved from https:\/\/arxiv.org\/abs\/1901.09330","journal-title":"arXiv:1901.09330"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3627822","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3627822","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:09Z","timestamp":1750178169000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3627822"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,19]]},"references-count":105,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3627822"],"URL":"https:\/\/doi.org\/10.1145\/3627822","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,19]]},"assertion":[{"value":"2022-09-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-25","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}