{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T06:57:13Z","timestamp":1790751433029,"version":"4.1.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2017,6,27]],"date-time":"2017-06-27T00:00:00Z","timestamp":1498521600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2017,6,30]]},"abstract":"<jats:p>Given a robust control system, physical simulation offers the potential for interactive human characters that move in realistic and responsive ways. In this article, we describe how to learn a scheduling scheme that reorders short control fragments as necessary at runtime to create a control system that can respond to disturbances and allows steering and other user interactions. These schedulers provide robust control of a wide range of highly dynamic behaviors, including walking on a ball, balancing on a bongo board, skateboarding, running, push-recovery, and breakdancing. We show that moderate-sized Q-networks can model the schedulers for these control tasks effectively and that those schedulers can be efficiently learned by the deep Q-learning algorithm.<\/jats:p>","DOI":"10.1145\/3083723","type":"journal-article","created":{"date-parts":[[2017,6,28]],"date-time":"2017-06-28T12:27:07Z","timestamp":1498652827000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":54,"title":["Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning"],"prefix":"10.1145","volume":"36","author":[{"given":"Libin","family":"Liu","sequence":"first","affiliation":[{"name":"Disney Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica","family":"Hodgins","sequence":"additional","affiliation":[{"name":"Disney Research and Carnegie Mellon University, Pittsburgh, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2017,6,27]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2011.62"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.325"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12321"},{"key":"e_1_2_2_4_1","volume-title":"Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.","author":"Bastien Fr\u00e9d\u00e9ric","year":"2012","unstructured":"Fr\u00e9d\u00e9ric Bastien , Pascal Lamblin , Razvan Pascanu , James Bergstra , Ian J. Goodfellow , Arnaud Bergeron , Nicolas Bouchard , and Yoshua Bengio . 2012 . Theano: New features and speed improvements . Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. Fr\u00e9d\u00e9ric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: New features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop."},{"key":"e_1_2_2_5_1","volume-title":"Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation.","author":"Bergstra James","year":"2010","unstructured":"James Bergstra , Olivier Breuleux , Fr\u00e9d\u00e9ric Bastien , Pascal Lamblin , Razvan Pascanu , Guillaume Desjardins , Joseph Turian , David Warde-Farley , and Yoshua Bengio . 2010 . Theano: A CPU and GPU math expression compiler . In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. James Bergstra, Olivier Breuleux, Fr\u00e9d\u00e9ric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: A CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation."},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201914)","author":"Buss Brian G.","unstructured":"Brian G. Buss , Alireza Ramezani , Kaveh Akbari Hamed , Brent A. Griffin , Kevin S. Galloway , and Jessy W. Grizzle . 2014. Preliminary walking experiments with underactuated 3D bipedal robot MARLO . In Proceedings of the 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201914) . 2529--2536. Brian G. Buss, Alireza Ramezani, Kaveh Akbari Hamed, Brent A. Griffin, Kevin S. Galloway, and Jessy W. Grizzle. 2014. Preliminary walking experiments with underactuated 3D bipedal robot MARLO. In Proceedings of the 2014 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201914). 2529--2536."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276384"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1618452.1618516"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1781156"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531388"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03189.x"},{"key":"e_1_2_2_12_1","unstructured":"Ga\u00ebl Guennebaud Beno\u00eet Jacob and others. 2010. Eigen v3. Retrieved from http:\/\/eigen.tuxfamily.org.  Ga\u00ebl Guennebaud Beno\u00eet Jacob and others. 2010. Eigen v3. Retrieved from http:\/\/eigen.tuxfamily.org."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2682626"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2767002"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2070781.2024197"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/1921427.1921447"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983616"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1781155"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1882261.1866160"},{"key":"e_1_2_2_20_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning (ICML\u201913)","author":"Levine Sergey","year":"2013","unstructured":"Sergey Levine and Vladlen Koltun . 2013 . Guided policy search . In Proceedings of the 30th International Conference on Machine Learning (ICML\u201913) . Sergey Levine and Vladlen Koltun. 2013. Guided policy search. In Proceedings of the 30th International Conference on Machine Learning (ICML\u201913)."},{"key":"e_1_2_2_21_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning (ICML\u201914)","author":"Levine Sergey","year":"2014","unstructured":"Sergey Levine and Vladlen Koltun . 2014 . Learning complex neural network policies with trajectory optimization . In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914) . Sergey Levine and Vladlen Koltun. 2014. Learning complex neural network policies with trajectory optimization. In Proceedings of the 31st International Conference on Machine Learning (ICML\u201914)."},{"key":"e_1_2_2_22_1","volume-title":"Continuous control with deep reinforcement learning. CoRR abs\/1509.02971","author":"Lillicrap Timothy P.","year":"2015","unstructured":"Timothy P. Lillicrap , Jonathan J. Hunt , Alexander Pritzel , Nicolas Heess , Tom Erez , Yuval Tassa , David Silver , and Daan Wierstra . 2015. Continuous control with deep reinforcement learning. CoRR abs\/1509.02971 ( 2015 ). http:\/\/arxiv.org\/abs\/1509.02971 Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. CoRR abs\/1509.02971 (2015). http:\/\/arxiv.org\/abs\/1509.02971"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0263574712000203"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2893476"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12571"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778865"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508427"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531386"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276385"},{"key":"e_1_2_2_30_1","volume-title":"Human-level control through deep reinforcement learning. Nature 518, 7540 (26","author":"Mnih Volodymyr","year":"2015","unstructured":"Volodymyr Mnih , Koray Kavukcuoglu , David Silver , Andrei A. Rusu , Joel Veness , Marc G. Bellemare , Alex Graves , Martin Riedmiller , Andreas K. Fidjeland , Georg Ostrovski , Stig Petersen , Charles Beattie , Amir Sadik , Ioannis Antonoglou , Helen King , Dharshan Kumaran , Daan Wierstra , Shane Legg , and Demis Hassabis . 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (26 Feb 2015 ), 529--533. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (26 Feb 2015), 529--533."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778808"},{"key":"e_1_2_2_32_1","volume-title":"Todorov","author":"Mordatch Igor","year":"2015","unstructured":"Igor Mordatch , Kendall Lowrey , Galen Andrew , Zoran Popovic , and Emanuel V . Todorov . 2015 . Interactive control of diverse complex characters with neural networks. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc ., 3114--3122. Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V. Todorov. 2015. Interactive control of diverse complex characters with neural networks. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc., 3114--3122."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185539"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531387"},{"key":"e_1_2_2_35_1","volume-title":"Deep Learning Workshop, International Conference on Machine Learning (ICML\u201915)","author":"Nair Arun","year":"2015","unstructured":"Arun Nair , Praveen Srinivasan , Sam Blackwell , Cagdas Alcicek , Rory Fearon , Alessandro De Maria , Vedavyas Panneershelvam , Mustafa Suleyman , Charles Beattie , Stig Petersen , Shane Legg , Volodymyr Mnih , Koray Kavukcuoglu , and David Silver . 2015 . Massively parallel methods for deep reinforcement learning . In Deep Learning Workshop, International Conference on Machine Learning (ICML\u201915) . Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. 2015. Massively parallel methods for deep reinforcement learning. In Deep Learning Workshop, International Conference on Machine Learning (ICML\u201915)."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2004.03.003"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766910"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925881"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276511"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2012.2210294"},{"key":"e_1_2_2_41_1","volume-title":"Proceedings of the 15th International Conference on Machine Learning (ICML\u201998)","author":"Sutton Richard S.","unstructured":"Richard S. Sutton , Doina Precup , and Satinder P. Singh . 1998. Intra-option learning about temporally abstract actions . In Proceedings of the 15th International Conference on Machine Learning (ICML\u201998) . 556--564. Richard S. Sutton, Doina Precup, and Satinder P. Singh. 1998. Intra-option learning about temporally abstract actions. In Proceedings of the 15th International Conference on Machine Learning (ICML\u201998). 556--564."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601121"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2011.30"},{"key":"e_1_2_2_44_1","unstructured":"Tijmen Tieleman and Geoff Hinton. 2012. Lecture 6.5\u2014RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.  Tijmen Tieleman and Geoff Hinton. 2012. Lecture 6.5\u2014RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276386"},{"key":"e_1_2_2_46_1","volume-title":"Deep reinforcement learning with double Q-learning. CoRR abs\/1509.06461","author":"van Hasselt Hado","year":"2015","unstructured":"Hado van Hasselt , Arthur Guez , and David Silver . 2015. Deep reinforcement learning with double Q-learning. CoRR abs\/1509.06461 ( 2015 ). http:\/\/arxiv.org\/abs\/1509.06461 Hado van Hasselt, Arthur Guez, and David Silver. 2015. Deep reinforcement learning with double Q-learning. CoRR abs\/1509.06461 (2015). http:\/\/arxiv.org\/abs\/1509.06461"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531366"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778810"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778811"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276509"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2330610"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3083723","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3083723","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:08Z","timestamp":1750217408000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3083723"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,27]]},"references-count":51,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2017,6,30]]}},"alternative-id":["10.1145\/3083723"],"URL":"https:\/\/doi.org\/10.1145\/3083723","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,6,27]]},"assertion":[{"value":"2016-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}