{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T18:15:15Z","timestamp":1775067315876,"version":"3.50.1"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T00:00:00Z","timestamp":1570665600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T00:00:00Z","timestamp":1570665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Reinforcement learning methods rely on rewards provided by the environment that are extrinsic to the agent. However, many real-world scenarios involve sparse or delayed rewards. In such cases, the agent can develop its own intrinsic reward function called curiosity to enable the agent to explore its environment in the quest of new skills. We propose a novel end-to-end curiosity mechanism for deep reinforcement learning methods, that allows an agent to gradually acquire new skills. Our method scales to high-dimensional problems, avoids the need of directly predicting the future, and, can perform in sequential decision scenarios. We formulate the curiosity as the ability of the agent to predict its own knowledge about the task. We base the prediction on the idea of skill learning to incentivize the discovery of new skills, and guide exploration towards promising solutions. To further improve data efficiency and generalization of the agent, we propose to learn a latent representation of the skills. We present a variety of sparse reward tasks in MiniGrid, MuJoCo, and Atari games. We compare the performance of an augmented agent that uses our curiosity reward to state-of-the-art learners. Experimental evaluation exhibits higher performance compared to reinforcement learning models that only learn by maximizing extrinsic rewards.<\/jats:p>","DOI":"10.1007\/s10994-019-05845-8","type":"journal-article","created":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T20:03:41Z","timestamp":1570737821000},"page":"493-512","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Skill-based curiosity for intrinsically motivated reinforcement learning"],"prefix":"10.1007","volume":"109","author":[{"given":"Nicolas","family":"Bougie","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ryutaro","family":"Ichise","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,10,10]]},"reference":[{"key":"5845_CR1","doi-asserted-by":"crossref","unstructured":"Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007). An application of reinforcement learning to aerobatic helicopter flight. In Proceedings of advances in neural information processing systems (pp. 1\u20138).","DOI":"10.7551\/mitpress\/7503.003.0006"},{"key":"5845_CR2","unstructured":"Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., & Welinder, P. et al. (2017). Hindsight experience replay. In Proceedings of advances in neural information processing systems."},{"issue":"1","key":"5845_CR3","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.robot.2012.05.008","volume":"61","author":"A Baranes","year":"2013","unstructured":"Baranes, A., & Oudeyer, P. Y. (2013). Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1), 49\u201373.","journal-title":"Robotics and Autonomous Systems"},{"key":"5845_CR4","unstructured":"Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. In Proceedings of advances in neural information processing systems (pp. 1471\u20131479)."},{"key":"5845_CR5","unstructured":"Burda, Y., Edwards, H., Storkey, A. J., & Klimov, O. (2018). Exploration by random network distillation. arXiv:1810.12894 ."},{"key":"5845_CR6","unstructured":"Chevalier-Boisvert, M., Willems, L., & Pal, S. (2018). Minimalistic gridworld environment for openai gym. https:\/\/github.com\/maximecb\/gym-minigrid ."},{"key":"5845_CR7","unstructured":"Eppe, M., Magg, S., & Wermter, S. (2018). Curriculum goal masking for continuous deep reinforcement learning. CoRR. arXiv:1809.06146 ."},{"key":"5845_CR8","doi-asserted-by":"crossref","unstructured":"Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2016). Deep spatial autoencoders for visuomotor learning. In Proceedings of the international conference on robotics and automation, IEEE (pp. 512\u2013519).","DOI":"10.1109\/ICRA.2016.7487173"},{"key":"5845_CR9","unstructured":"Florensa, C., Held, D., Wulfmeier, M., Zhang, M., & Abbeel, P. (2017). Reverse curriculum generation for reinforcement learning. arXiv preprint arXiv:1707.05300 ."},{"key":"5845_CR10","unstructured":"Held, D., Geng, X., Florensa, C., & Abbeel, P. (2017). Automatic goal generation for reinforcement learning agents. arXiv preprint arXiv:1705.06366 ."},{"key":"5845_CR11","unstructured":"Itti, L., & Baldi, P. F. (2006). Bayesian surprise attracts human attention. In Proceedings of advances in neural information processing systems (pp. 547\u2013554)."},{"key":"5845_CR12","unstructured":"Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2017). Reinforcement learning with unsupervised auxiliary tasks. In International conference on learning representations."},{"key":"5845_CR13","unstructured":"Kingma, D. P., & Ba, J. (2015). Adam: a method for stochastic optimization. In Proceedings of the international conference on learning representations."},{"key":"5845_CR14","unstructured":"Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. In Proceedings of the international conference on learning representations."},{"key":"5845_CR15","unstructured":"Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097\u20131105)."},{"issue":"2","key":"5845_CR16","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1162\/EVCO_a_00025","volume":"19","author":"J Lehman","year":"2011","unstructured":"Lehman, J., & Stanley, K. O. (2011). Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2), 189\u2013223.","journal-title":"Evolutionary Computation"},{"issue":"1","key":"5845_CR17","first-page":"1334","volume":"17","author":"S Levine","year":"2016","unstructured":"Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016). End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1), 1334\u20131373.","journal-title":"The Journal of Machine Learning Research"},{"key":"5845_CR18","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., & Wierstra, D. et al. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 ."},{"issue":"7540","key":"5845_CR19","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529.","journal-title":"Nature"},{"key":"5845_CR20","unstructured":"Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., & Harley, T. et al. (2016). Asynchronous methods for deep reinforcement learning. In Proceedings of the international conference on machine learning."},{"key":"5845_CR21","unstructured":"Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., & Levine, S. (2018). Visual reinforcement learning with imagined goals. In Proceedings of advances in neural information processing systems (pp. 9209\u20139220)."},{"key":"5845_CR22","unstructured":"Ng, A. Y., & Russell, S. (2000). Algorithms for inverse reinforcement learning. In Proceedings of the seventeen international conference on machine learning, Citeseer."},{"key":"5845_CR23","first-page":"278","volume":"99","author":"AY Ng","year":"1999","unstructured":"Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. Proceedings of the International Conference on Machine Learning, 99, 278\u2013287.","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"5845_CR24","unstructured":"Ostrovski, G., Bellemare, M. G., van\u00a0den Oord, A., & Munos, R. (2017). Count-based exploration with neural density models. In Proceedings of the international conference on machine learning (pp. 2721\u20132730)."},{"key":"5845_CR25","doi-asserted-by":"crossref","unstructured":"Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. In Proceedings of the international conference on international conference on machine learning.","DOI":"10.1109\/CVPRW.2017.70"},{"key":"5845_CR26","unstructured":"Racani\u00e8re, S., Weber, T., Reichert, D., Buesing, L., Guez, A., & Jimenez\u00a0Rezende, D. et al. (2017). Imagination-augmented agents for deep reinforcement learning. In Proceedings of advances in neural information processing systems."},{"key":"5845_CR27","unstructured":"Rezende, D.J., Mohamed, S., Danihelka, I., Gregor, K., & Wierstra, D. (2016). One-shot generalization in deep generative models. In Proceedings of the international conference on international conference on machine learning (pp. 1521\u20131529)."},{"issue":"5","key":"5845_CR28","doi-asserted-by":"publisher","first-page":"2789","DOI":"10.3390\/e16052789","volume":"16","author":"C Salge","year":"2014","unstructured":"Salge, C., Glackin, C., & Polani, D. (2014). Changing the environment based on empowerment as intrinsic motivation. Entropy, 16(5), 2789\u20132819.","journal-title":"Entropy"},{"key":"5845_CR29","unstructured":"Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., & Gelly, S. (2019). Episodic curiosity through reachability. In Proceedings of the international conference on learning representations."},{"key":"5845_CR30","unstructured":"Schaul, T., Horgan, D., Gregor, K., & Silver, D. (2015). Universal value function approximators. In Proceedings of the international conference on machine learning."},{"key":"5845_CR31","doi-asserted-by":"crossref","unstructured":"Schmidhuber, J. (1991a). Curious model-building control systems. In Proceedings of the IEEE international joint conference on neural networks, IEEE.","DOI":"10.1109\/IJCNN.1991.170605"},{"key":"5845_CR32","doi-asserted-by":"crossref","unstructured":"Schmidhuber, J. (1991b). A possibility for implementing curiosity and boredom in model-building neural controllers. In Proceedings of the international conference on simulation of adaptive behavior: From animals to animats (pp. 222\u2013227).","DOI":"10.7551\/mitpress\/3115.003.0030"},{"key":"5845_CR33","first-page":"1889","volume":"37","author":"J Schulman","year":"2015","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., & Moritz, P. (2015). Trust region policy optimization. Proceedings of the International Conference on International Conference on Machine Learning, 37, 1889\u20131897.","journal-title":"Proceedings of the International Conference on International Conference on Machine Learning"},{"key":"5845_CR34","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 ."},{"key":"5845_CR35","unstructured":"Stadie, B. C., Levine, S., & Abbeel, P. (2015). Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814 ."},{"key":"5845_CR36","unstructured":"Stanton, C., & Clune, J. (2019). Deep curiosity search: Intra-life exploration improves performance on challenging deep reinforcement learning problems. In Proceedings of the international conference on international conference on machine learning."},{"issue":"1","key":"5845_CR37","first-page":"9","volume":"3","author":"RS Sutton","year":"1988","unstructured":"Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9\u201344.","journal-title":"Machine Learning"},{"key":"5845_CR38","volume-title":"Reinforcement learning: An introduction","author":"RS Sutton","year":"1998","unstructured":"Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press."},{"key":"5845_CR39","unstructured":"Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems (pp. 1057\u20131063)."},{"key":"5845_CR40","doi-asserted-by":"crossref","unstructured":"Todorov, E., Erez, T., & Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In Proceedings of the international conference on intelligent robots and systems (pp. 5026\u20135033).","DOI":"10.1109\/IROS.2012.6386109"},{"key":"5845_CR41","unstructured":"Wang, Z., Schaul, T., Hessel, M., Van\u00a0Hasselt, H., Lanctot, M., & De\u00a0Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. In Proceedings of the international conference on international conference on machine learning."},{"key":"5845_CR42","unstructured":"Zhao, S., Ren, H., Yuan, A., Song, J., Goodman, N., & Ermon, S. (2018). Bias and generalization in deep generative models: An empirical study. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds). Proceedings of advances in neural information processing systems (pp. 10792\u201310801)."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-019-05845-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10994-019-05845-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-019-05845-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T21:09:29Z","timestamp":1721855369000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10994-019-05845-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,10]]},"references-count":42,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["5845"],"URL":"https:\/\/doi.org\/10.1007\/s10994-019-05845-8","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,10]]},"assertion":[{"value":"26 April 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2019","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 September 2019","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 October 2019","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}