{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T05:57:25Z","timestamp":1790402245270,"version":"4.1.0"},"reference-count":39,"publisher":"National Academy of Sciences","issue":"48","license":[{"start":{"date-parts":[[2021,2,17]],"date-time":"2021-02-17T00:00:00Z","timestamp":1613520000000},"content-version":"vor","delay-in-days":184,"URL":"https:\/\/www.pnas.org\/site\/aboutpnas\/licenses.xhtml"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p>The combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decision-making problems that are currently intractable. One obstacle to overcome is the amount of data needed by learning systems of this type. In this article, we propose to address this issue through a divide-and-conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, this problem decomposition can be seamlessly accommodated within the standard reinforcement-learning formalism. The specific way we do so is through a generalization of two fundamental operations in reinforcement learning: policy improvement and policy evaluation. The generalized version of these operations allow one to leverage the solution of some tasks to speed up the solution of others. If the reward function of a task can be well approximated as a linear combination of the reward functions of tasks previously solved, we can reduce a reinforcement-learning problem to a simpler linear regression. When this is not the case, the agent can still exploit the task solutions by using them to interact with and learn about the environment. Both strategies considerably reduce the amount of data needed to solve a reinforcement-learning problem.<\/jats:p>","DOI":"10.1073\/pnas.1907370117","type":"journal-article","created":{"date-parts":[[2020,8,17]],"date-time":"2020-08-17T20:54:01Z","timestamp":1597697641000},"page":"30079-30087","update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":63,"title":["Fast reinforcement learning with generalized policy updates"],"prefix":"10.1073","volume":"117","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6168-6972","authenticated-orcid":false,"given":"Andr\u00e9","family":"Barreto","sequence":"first","affiliation":[{"name":"DeepMind, London EC4A 3TW, United Kingdom;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6181-5452","authenticated-orcid":false,"given":"Shaobo","family":"Hou","sequence":"additional","affiliation":[{"name":"DeepMind, London EC4A 3TW, United Kingdom;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diana","family":"Borsa","sequence":"additional","affiliation":[{"name":"DeepMind, London EC4A 3TW, United Kingdom;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Silver","sequence":"additional","affiliation":[{"name":"DeepMind, London EC4A 3TW, United Kingdom;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Doina","family":"Precup","sequence":"additional","affiliation":[{"name":"DeepMind, London EC4A 3TW, United Kingdom;"},{"name":"School of Computer Science, McGill University, Montreal, QC H3A 0E9, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2020,8,17]]},"reference":[{"key":"e_1_3_4_1_2","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton R. S.","year":"2018","unstructured":"R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, 2018)."},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913495721"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.275.5306.1593"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/nn1309"},{"key":"e_1_3_4_5_2","first-page":"1","volume-title":"IEEE International Conference on Rehabilitation Robotics","author":"Pilarski P. M.","year":"2011","unstructured":"P. M. Pilarski , \u201cOnline human training of a myoelectric prosthesis controller via actor-critic reinforcement learning\u201d in IEEE International Conference on Rehabilitation Robotics (IEEE, 2011), pp. 1\u20137."},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/203330.203343"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1259433"},{"key":"e_1_3_4_8_2","first-page":"463","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Randl\u00f8v J.","year":"1998","unstructured":"J. Randl\u00f8v, P. Alstr\u00f8m, \u201cLearning to drive a bicycle using reinforcement learning and shaping\u201d in Proceedings of the International Conference on Machine Learning (ICML) (Morgan Kaufmann Publishers, Inc., 1998), pp. 463\u2013471."},{"key":"e_1_3_4_9_2","first-page":"799","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Ng A. Y.","year":"2003","unstructured":"A. Y. Ng, H. J. Kim, M. I. Jordan, S. Sastry, \u201cAutonomous helicopter flight via reinforcement learning\u201d in Advances in Neural Information Processing Systems (NIPS) (MIT Press, 2003), pp. 799\u2013806."},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aar6404"},{"key":"e_1_3_4_14_2","volume-title":"AAAI Spring Symposium Series","author":"Tsividis P.","year":"2017","unstructured":"P. Tsividis, T. Pouncy, J. Xu, J. Tenenbaum, S. Gershman, \u201cHuman learning in Atari\u201d in AAAI Spring Symposium Series (AAAI Press, 2017)."},{"key":"e_1_3_4_15_2","volume-title":"The Behavior of Organisms: An Experimental Analysis","author":"Skinner B. F.","year":"1938","unstructured":"B. F. Skinner, The Behavior of Organisms: An Experimental Analysis (Appleton-Century, 1938)."},{"key":"e_1_3_4_16_2","volume-title":"Principles of Behavior","author":"Hull C. L.","year":"1943","unstructured":"C. L. Hull, Principles of Behavior (Appleton-Century, New York, NY, 1943)."},{"key":"e_1_3_4_17_2","first-page":"663","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Ng A.","year":"2000","unstructured":"A. Ng, S. Russell, \u201cAlgorithms for inverse reinforcement learning\u201d in Proceedings of the International Conference on Machine Learning (ICML) (Morgan Kaufmann Publishers, Inc., 2000), pp. 663\u2013670."},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316887"},{"key":"e_1_3_4_19_2","volume-title":"Dynamic Programming","author":"Bellman R. E.","year":"1957","unstructured":"R. E. Bellman, Dynamic Programming (Princeton University Press, 1957)."},{"key":"e_1_3_4_20_2","volume-title":"Neuro-Dynamic Programming","author":"Bertsekas D. P.","year":"1996","unstructured":"D. P. Bertsekas, J. N. Tsitsiklis, Neuro-Dynamic Programming (Athena Scientific, 1996)."},{"key":"e_1_3_4_21_2","first-page":"560","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Munos R.","year":"2003","unstructured":"R. Munos, \u201cError bounds for approximate policy iteration\u201d in Proceedings of the International Conference on Machine Learning (ICML) (AAAI Press, 2003), pp. 560\u2013567."},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00115009"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"e_1_3_4_24_2","volume-title":"Dynamic Programming and Markov Processes","author":"Howard R.","year":"1960","unstructured":"R. Howard, Dynamic Programming and Markov Processes (MIT Press, 1960)."},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1994.6.6.1185"},{"key":"e_1_3_4_26_2","first-page":"761","volume-title":"International Joint Conference on Autonomous Agents and Multiagent Systems","author":"Sutton R. S.","year":"2011","unstructured":"R. S. Sutton , \u201cHorde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction\u201d in International Joint Conference on Autonomous Agents and Multiagent Systems (International Foundation for Autonomous Agents and Multiagent Systems, 2011), pp. 761\u2013768."},{"key":"e_1_3_4_27_2","first-page":"1312","volume-title":"International Conference on Machine Learning (ICML)","author":"Schaul T.","year":"2015","unstructured":"T. Schaul, D. Horgan, K. Gregor, D. Silver, \u201cUniversal value function approximators\u201d in International Conference on Machine Learning (ICML) (PMLR, 2015), vol. 37, pp. 1312\u20131320."},{"key":"e_1_3_4_28_2","first-page":"4055","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Barreto A.","year":"2017","unstructured":"A. Barreto , \u201cSuccessor features for transfer in reinforcement learning\u201d in Advances in Neural Information Processing Systems (NIPS) (Curran Associates, Inc., 2017), pp. 4055\u20134065."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1993.5.4.613"},{"key":"e_1_3_4_30_2","first-page":"501","volume-title":"Proceedings of the International Conference on Machine","author":"Barreto A.","year":"2018","unstructured":"A. Barreto , \u201cTransfer in deep reinforcement learning using successor features and generalised policy improvement\u201d in Proceedings of the International Conference on Machine (PMLR, 2018), vol. 80, pp. 501\u2013510."},{"key":"e_1_3_4_31_2","first-page":"13052","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Barreto A.","year":"2019","unstructured":"A. Barreto , \u201cThe option keyboard: Combining skills in reinforcement learning\u201d in Advances in Neural Information Processing Systems (NeurIPS) (Curran Associates, Inc., 2019), pp. 13052\u201313062."},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1025696116075"},{"key":"e_1_3_4_33_2","volume-title":"Deep Learning","author":"Goodfellow I.","year":"2016","unstructured":"I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, 2016)."},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007379606734"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(99)00052-1"},{"key":"e_1_3_4_36_2","first-page":"640","volume-title":"Advances in Neural Information Processing Systems (NIPS)","author":"Thrun S.","year":"1996","unstructured":"S. Thrun, \u201cIs learning the N-th thing any easier than learning the first?\u201d in Advances in Neural Information Processing Systems (NIPS) (MIT Press, 1996), pp. 640\u2013646."},{"key":"e_1_3_4_37_2","unstructured":"D. Borsa \u201cUniversal successor features approximators\u201d in International Conference on Learning Representations (ICLR) (2019). https:\/\/openreview.net\/forum?id=S1VWjiRcKX. Accessed 5 August 2020."},{"key":"e_1_3_4_38_2","first-page":"2911","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Hunt J.","year":"2019","unstructured":"J. Hunt, A. Barreto, T. Lillicrap, N. Heess, \u201cComposing entropic policies using divergence correction\u201d in Proceedings of the International Conference on Machine Learning (ICML) (PMLR, 2019), vol. 97, pp. 2911\u20132920."},{"key":"e_1_3_4_39_2","unstructured":"S. Hansen \u201cFast task inference with variational intrinsic successor features\u201d in International Conference on Learning Representations (ICLR) (2020). https:\/\/openreview.net\/forum?id=BJeAHkrYDS. Accessed 5 August 2020."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.1907370117","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T05:22:37Z","timestamp":1649827357000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/full\/10.1073\/pnas.1907370117"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,17]]},"references-count":39,"journal-issue":{"issue":"48","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["10.1073\/pnas.1907370117"],"URL":"https:\/\/doi.org\/10.1073\/pnas.1907370117","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,17]]},"assertion":[{"value":"2020-08-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}