{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T16:59:44Z","timestamp":1777568384095,"version":"3.51.4"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100014553","name":"Samsung Advanced Institute of Technology","doi-asserted-by":"publisher","award":["IO200317-07173-01"],"award-info":[{"award-number":["IO200317-07173-01"]}],"id":[{"id":"10.13039\/100014553","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>Reinforcement learning (RL) has been proven effective in many scenarios, including environment exploration and motion planning. However, its application in data-driven character control has produced relatively simple motion results compared to recent approaches that have used large complex motion data without RL. In this paper, we provide a real-time motion control method that can generate high-quality and complex motion results from various sets of unstructured data while retaining the advantage of using RL, which is the discovery of optimal behaviors by trial and error. We demonstrate the results for a character achieving different tasks, from simple direction control to complex avoidance of moving obstacles. Our system works equally well on biped\/quadruped characters, with motion data ranging from 1 to 48 minutes, without any manual intervention. To achieve this, we exploit a finite set of discrete actions, where each action represents full-body future motion features. We first define a subset of actions that can be selected in each state and store these pieces of information in databases during the preprocessing step. The use of this subset of actions enables the effective learning of control policy even from a large set of motion data. To achieve interactive performance at run-time, we adopt a proposal network and a k-nearest neighbor action sampler.<\/jats:p>","DOI":"10.1145\/3478513.3480512","type":"journal-article","created":{"date-parts":[[2021,12,10]],"date-time":"2021-12-10T18:28:45Z","timestamp":1639160925000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Motion recommendation for online character control"],"prefix":"10.1145","volume":"40","author":[{"given":"Kyungmin","family":"Cho","sequence":"first","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chaelin","family":"Kim","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jungjin","family":"Park","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joonkyu","family":"Park","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junyong","family":"Noh","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,12,10]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Deep Reinforcement Learning for Navigation in AAA Video Games. arXiv preprint arXiv:2011.04764","author":"Alonso Eloi","year":"2020","unstructured":"Eloi Alonso , Maxim Peter , David Goumard , and Joshua Romoff . 2020. Deep Reinforcement Learning for Navigation in AAA Video Games. arXiv preprint arXiv:2011.04764 ( 2020 ). Eloi Alonso, Maxim Peter, David Goumard, and Joshua Romoff. 2020. Deep Reinforcement Learning for Navigation in AAA Video Games. arXiv preprint arXiv:2011.04764 (2020)."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/566654.566606"},{"key":"e_1_2_2_3_1","volume-title":"Miles Brundage, and Anil Anthony Bharath.","author":"Arulkumaran Kai","year":"2017","unstructured":"Kai Arulkumaran , Marc Peter Deisenroth , Miles Brundage, and Anil Anthony Bharath. 2017 . A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 (2017). Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 (2017)."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356536"},{"key":"e_1_2_2_5_1","volume-title":"Proc. of GDC 2018","author":"Bollo David","year":"2016","unstructured":"David Bollo . 2016 . Inertialization: High-Performance Animation Transitions in'Gears of War '. Proc. of GDC 2018 (2016). David Bollo. 2016. Inertialization: High-Performance Animation Transitions in'Gears of War'. Proc. of GDC 2018 (2016)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3084363.3085069"},{"key":"e_1_2_2_7_1","volume-title":"Openai gym. arXiv preprint arXiv:1606.01540","author":"Brockman Greg","year":"2016","unstructured":"Greg Brockman , Vicki Cheung , Ludwig Pettersson , Jonas Schneider , John Schulman , Jie Tang , and Wojciech Zaremba . 2016. Openai gym. arXiv preprint arXiv:1606.01540 ( 2016 ). Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016)."},{"key":"e_1_2_2_8_1","volume-title":"Proc. of Nucl. ai","author":"B\u00fcttner Michael","year":"2015","unstructured":"Michael B\u00fcttner and Simon Clavet . 2015 . Motion Matching-The Road to Next Gen Animation . Proc. of Nucl. ai (2015). Michael B\u00fcttner and Simon Clavet. 2015. Motion Matching-The Road to Next Gen Animation. Proc. of Nucl. ai (2015)."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274247.3274506"},{"key":"e_1_2_2_10_1","volume-title":"Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679","author":"Dulac-Arnold Gabriel","year":"2015","unstructured":"Gabriel Dulac-Arnold , Richard Evans , Hado van Hasselt , Peter Sunehag , Timothy Lillicrap , Jonathan Hunt , Timothy Mann , Theophane Weber , Thomas Degris , and Ben Coppin . 2015. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679 ( 2015 ). Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2015. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679 (2015)."},{"key":"e_1_2_2_11_1","volume-title":"Robust Motion In-Betweening. 39, 4","author":"Harvey F\u00e9lix G.","year":"2020","unstructured":"F\u00e9lix G. Harvey , Mike Yurick , Derek Nowrouzezahrai , and Christopher Pal . 2020. Robust Motion In-Betweening. 39, 4 ( 2020 ). F\u00e9lix G. Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. 2020. Robust Motion In-Betweening. 39, 4 (2020)."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1230100.1230123"},{"key":"e_1_2_2_13_1","unstructured":"Nicolas Heess Dhruva TB Srinivasan  Sriram Jay Lemmon Josh Merel Greg Wayne Yuval Tassa Tom Erez Ziyu Wang SM Eslami etal 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).  Nicolas Heess Dhruva TB Srinivasan Sriram Jay Lemmon Josh Merel Greg Wayne Yuval Tassa Tom Erez Ziyu Wang SM Eslami et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017)."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392440"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073663"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925975"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2820903.2820918"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322966"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1186562.1015760"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/566654.566605"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/566654.566607"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.gmod.2005.03.004"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322972"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1661412.1618515"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866158.1866160"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1966394.1966402"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185524"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392422"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2893476"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1833349.1778865"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392433"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276385"},{"key":"e_1_2_2_33_1","volume-title":"Hierarchical Visuomotor Control of Humanoids. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJfYvo09Y7","author":"Merel Josh","year":"2019","unstructured":"Josh Merel , Arun Ahuja , Vu Pham , Saran Tunyasuvunakool , Siqi Liu , Dhruva Tirumala , Nicolas Heess , and Greg Wayne . 2019 a. Hierarchical Visuomotor Control of Humanoids. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJfYvo09Y7 Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. 2019a. Hierarchical Visuomotor Control of Humanoids. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJfYvo09Y7"},{"key":"e_1_2_2_34_1","volume-title":"Neural Probabilistic Motor Primitives for Humanoid Control. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJl6TjRcY7","author":"Merel Josh","year":"2019","unstructured":"Josh Merel , Leonard Hasenclever , Alexandre Galashov , Arun Ahuja , Vu Pham , Greg Wayne , Yee Whye Teh , and Nicolas Heess . 2019 b. Neural Probabilistic Motor Primitives for Humanoid Control. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJl6TjRcY7 Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. 2019b. Neural Probabilistic Motor Primitives for Humanoid Control. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=BJl6TjRcY7"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2366145.2366172"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1640443.1640452"},{"key":"e_1_2_2_37_1","volume-title":"Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602","author":"Mnih Volodymyr","year":"2013","unstructured":"Volodymyr Mnih , Koray Kavukcuoglu , David Silver , Alex Graves , Ioannis Antonoglou , Daan Wierstra , and Martin Riedmiller . 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 ( 2013 ). Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)."},{"key":"e_1_2_2_38_1","volume-title":"Neural discrete representation learning. arXiv preprint arXiv:1711.00937","author":"van den Oord Aaron","year":"2017","unstructured":"Aaron van den Oord , Oriol Vinyals , and Koray Kavukcuoglu . 2017. Neural discrete representation learning. arXiv preprint arXiv:1711.00937 ( 2017 ). Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. arXiv preprint arXiv:1711.00937 (2017)."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356501"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201311"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925881"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073602"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276510"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356505"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392450"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276386"},{"key":"e_1_2_2_47_1","volume-title":"Q-Learning in enormous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116","author":"de Wiele Tom Van","year":"2020","unstructured":"Tom Van de Wiele , David Warde-Farley , Andriy Mnih , and Volodymyr Mnih . 2020. Q-Learning in enormous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116 ( 2020 ). Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. 2020. Q-Learning in enormous action spaces via amortized approximate maximization. arXiv preprint arXiv:2001.08116 (2020)."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.5555\/3016100.3016191"},{"key":"e_1_2_2_49_1","volume-title":"UniCon: Universal Neural Controller For Physics-based Character Motion. arXiv preprint arXiv:2011.15119","author":"Wang Tingwu","year":"2020","unstructured":"Tingwu Wang , Yunrong Guo , Maria Shugrina , and Sanja Fidler . 2020. UniCon: Universal Neural Controller For Physics-based Character Motion. arXiv preprint arXiv:2011.15119 ( 2020 ). Tingwu Wang, Yunrong Guo, Maria Shugrina, and Sanja Fidler. 2020. UniCon: Universal Neural Controller For Physics-based Character Motion. arXiv preprint arXiv:2011.15119 (2020)."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356499"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130833"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275023"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14115"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201397"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201366"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.gmod.2009.04.001"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3478513.3480512","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3478513.3480512","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:11:49Z","timestamp":1750191109000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3478513.3480512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":56,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3478513.3480512"],"URL":"https:\/\/doi.org\/10.1145\/3478513.3480512","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2021-12-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}