{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T09:58:58Z","timestamp":1777715938691,"version":"3.51.4"},"reference-count":49,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2022,4,1]],"date-time":"2022-04-01T00:00:00Z","timestamp":1648771200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001655","name":"Deutscher Akademischer Austauschdienst","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001655","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["FA8750-18-C-0101"],"award-info":[{"award-number":["FA8750-18-C-0101"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"publisher","award":["FA9550-17-1-0308"],"award-info":[{"award-number":["FA9550-17-1-0308"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007297","name":"Office of Naval Research Global","doi-asserted-by":"publisher","award":["20-1-2736"],"award-info":[{"award-number":["20-1-2736"]}],"id":[{"id":"10.13039\/100007297","id-type":"DOI","asserted-by":"publisher"}]},{"name":"CONIX Research Center"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2022,4]]},"abstract":"<jats:p>Reward learning enables robots to learn adaptable behaviors from human input. Traditional methods model the reward as a linear function of hand-crafted features, but that requires specifying all the relevant features a priori, which is impossible for real-world tasks. To get around this issue, recent deep Inverse Reinforcement Learning (IRL) methods learn rewards directly from the raw state but this is challenging because the robot has to implicitly learn the features that are important and how to combine them, simultaneously. Instead, we propose a divide-and-conquer approach: focus human input specifically on learning the features separately, and only then learn how to combine them into a reward. We introduce a novel type of human input for teaching features and an algorithm that utilizes it to learn complex features from the raw state space. The robot can then learn how to combine them into a reward using demonstrations, corrections, or other reward learning frameworks. We demonstrate our method in settings where all features have to be learned from scratch, as well as where some of the features are known. By first focusing human input specifically on the feature(s), our method decreases sample complexity and improves generalization of the learned reward over a deep IRL baseline. We show this in experiments with a physical 7-DoF robot manipulator, and in a user study conducted in a simulated environment.<\/jats:p>","DOI":"10.1177\/02783649221078031","type":"journal-article","created":{"date-parts":[[2022,4,15]],"date-time":"2022-04-15T23:15:52Z","timestamp":1650064552000},"page":"497-518","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":18,"title":["Inducing structure in reward learning by learning features"],"prefix":"10.1177","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9507-7427","authenticated-orcid":false,"given":"Andreea","family":"Bobu","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marius","family":"Wiggert","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Claire","family":"Tomlin","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anca D","family":"Dragan","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2022,4,15]]},"reference":[{"key":"bibr1-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015430"},{"key":"bibr2-02783649221078031","unstructured":"Amodei D, Clark J (2016) Faulty reward functions in the wild. https:\/\/blog.openai.com\/faulty-reward-functions\/."},{"key":"bibr3-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"bibr4-02783649221078031","unstructured":"Arjovsky M, Bottou L, Gulrajani I, et al. (2019) Invariant risk minimization. ArXiv abs\/1907.02893."},{"key":"bibr5-02783649221078031","first-page":"217","volume-title":"Proceedings of the 1st Annual Conference on Robot Learning, Proceedings of Machine Learning Research","author":"Bajcsy A","year":"2017"},{"key":"bibr6-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171267"},{"key":"bibr7-02783649221078031","unstructured":"Baker C, B Tenenbaum J, R Saxe R (2007) Goal inference as inverse planning. In: Proceedings of the 29th Annual Conference of the Cognitive Science Society."},{"key":"bibr8-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2020.2971415"},{"key":"bibr9-02783649221078031","first-page":"796","volume-title":"Proceedings of The 2nd Conference on Robot Learning, Proceedings of Machine Learning Research","author":"Bobu A","year":"2018"},{"key":"bibr10-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1145\/3434073.3444667"},{"key":"bibr11-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v29i4.2203"},{"key":"bibr12-02783649221078031","first-page":"1165","volume-title":"Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research","author":"Brown D","year":"2020"},{"key":"bibr13-02783649221078031","first-page":"783","volume-title":"International Conference on Machine Learning","author":"Brown D","year":"2019"},{"key":"bibr14-02783649221078031","first-page":"362","volume-title":"Conference on Robot Learning","author":"Brown DS","year":"2018"},{"key":"bibr15-02783649221078031","first-page":"691","volume":"12","author":"Choi J","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"bibr16-02783649221078031","unstructured":"Choi J, Kim KE (2013) Bayesian nonparametric feature construction for inverse reinforcement learning. In: Twenty-Third International Joint Conference on Artificial Intelligence."},{"key":"bibr17-02783649221078031","volume-title":"Advances in Neural Information Processing Systems","volume":"30","author":"Christiano PF","year":"2017"},{"key":"bibr18-02783649221078031","unstructured":"Coumans E, Bai Y (2016\u20132019) Pybullet, a python module for physics simulation for games, robotics and machine learning. http:\/\/pybullet.org."},{"key":"bibr19-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139510"},{"key":"bibr20-02783649221078031","first-page":"49","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML\u201916","author":"Finn C","year":"2016"},{"key":"bibr21-02783649221078031","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2018.XIV.069"},{"key":"bibr22-02783649221078031","author":"Fridovich-Keil D","year":"2019","journal-title":"International Journal of Robotics Research"},{"key":"bibr23-02783649221078031","unstructured":"Fu J, Luo K, Levine S (2018a) Learning robust rewards with adverserial inverse reinforcement learning. In: International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rkHywl-A-"},{"key":"bibr24-02783649221078031","unstructured":"Fu J, Singh A, Ghosh D, et al. (2018b) Variational inverse control with events: A general framework for data-driven reward definition. arXiv preprint arXiv:1805.11686."},{"key":"bibr25-02783649221078031","volume-title":"Advances in Neural Information Processing Systems","volume":"30","author":"Hadfield-Menell D","year":"2017"},{"key":"bibr26-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"bibr27-02783649221078031","first-page":"8464","volume-title":"Advances in Neural Information Processing Systems","author":"Haug L","year":"2018"},{"key":"bibr28-02783649221078031","first-page":"8011","volume-title":"Advances in Neural Information Processing Systems","volume":"31","author":"Ibarz B","year":"2018"},{"key":"bibr29-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1177\/0278364915581193"},{"key":"bibr30-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918776060"},{"key":"bibr31-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRev.106.620"},{"key":"bibr32-02783649221078031","first-page":"1342","volume-title":"Advances in Neural Information Processing Systems","author":"Levine S","year":"2010"},{"key":"bibr33-02783649221078031","first-page":"19","volume-title":"Advances in Neural Information Processing Systems","author":"Levine S","year":"2011"},{"key":"bibr34-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04174-7_3"},{"key":"bibr35-02783649221078031","doi-asserted-by":"publisher","DOI":"10.2460\/ajvr.67.2.323"},{"key":"bibr36-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1561\/2300000053"},{"key":"bibr37-02783649221078031","first-page":"1153","volume-title":"Advances in Neural Information Processing Systems","author":"Ratliff N","year":"2007"},{"key":"bibr38-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143936"},{"key":"bibr39-02783649221078031","volume-title":"ICML","author":"Reddy S","year":"2020"},{"key":"bibr40-02783649221078031","unstructured":"Reddy S, Dragan AD, Levine S (2020b) SQIL: imitation learning via reinforcement learning with sparse rewards. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 26\u201330 April 2020. https:\/\/openreview.net\/forum?id=S1xKd24twB."},{"key":"bibr41-02783649221078031","unstructured":"Russell S, Norvig P (2002) Artificial intelligence: a modern approach."},{"key":"bibr42-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2016.7759036"},{"key":"bibr43-02783649221078031","first-page":"1","volume-title":"Robotics: Science and Systems","volume":"9","author":"Schulman J","year":"2013"},{"key":"bibr44-02783649221078031","volume-title":"The Nature of Statistical Learning Theory","author":"Vapnik V","year":"2013"},{"key":"bibr45-02783649221078031","first-page":"575","volume-title":"Advances in Neural Information Processing Systems","author":"Vernaza P","year":"2012"},{"key":"bibr46-02783649221078031","volume-title":"Theory of Games and Economic Behavior","author":"Von Neumann J","year":"1945"},{"key":"bibr47-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2016.7759328"},{"key":"bibr48-02783649221078031","first-page":"1433","volume-title":"Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3, AAAI\u201908","author":"Ziebart BD","year":"2008"},{"key":"bibr49-02783649221078031","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561839"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649221078031","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02783649221078031","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649221078031","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649221078031","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:16:55Z","timestamp":1777457815000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02783649221078031"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4]]},"references-count":49,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,4]]}},"alternative-id":["10.1177\/02783649221078031"],"URL":"https:\/\/doi.org\/10.1177\/02783649221078031","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4]]}}}