{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T06:02:34Z","timestamp":1784268154122,"version":"3.55.0"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T00:00:00Z","timestamp":1701734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,12,5]]},"abstract":"<jats:p>Recent advances in learning reusable motion priors have demonstrated their effectiveness in generating naturalistic behaviors. In this paper, we propose a new learning framework in this paradigm for controlling physics-based characters with improved motion quality and diversity over existing methods. The proposed method uses reinforcement learning (RL) to initially track and imitate life-like movements from unstructured motion clips using the discrete information bottleneck, as adopted in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure compresses the most relevant information from the motion clips into a compact yet informative latent space, i.e., a discrete space over vector quantized codes. By sampling codes in the space from a trained categorical prior distribution, high-quality life-like behaviors can be generated, similar to the usage of VQ-VAE in computer vision. Although this prior distribution can be trained with the supervision of the encoder's output, it follows the original motion clip distribution in the dataset and could lead to imbalanced behaviors in our setting. To address the issue, we further propose a technique named prior shifting to adjust the prior distribution using curiosity-driven RL. The outcome distribution is demonstrated to offer sufficient behavioral diversity and significantly facilitates upper-level policy learning for downstream tasks. We conduct comprehensive experiments using humanoid characters on two challenging downstream tasks, sword-shield striking and two-player boxing game. Our results demonstrate that the proposed framework is capable of controlling the character to perform considerably high-quality movements in terms of behavioral strategies, diversity, and realism. Videos, codes, and data are available at https:\/\/tencent-roboticsx.github.io\/NCP\/.<\/jats:p>","DOI":"10.1145\/3618397","type":"journal-article","created":{"date-parts":[[2023,12,5]],"date-time":"2023-12-05T10:20:48Z","timestamp":1701771648000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":44,"title":["Neural Categorical Priors for Physics-Based Character Control"],"prefix":"10.1145","volume":"42","author":[{"given":"Qingxu","family":"Zhu","sequence":"first","affiliation":[{"name":"Tencent Robotics X, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"He","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tencent Robotics X, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mengting","family":"Lan","sequence":"additional","affiliation":[{"name":"XVERSE, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Han","sequence":"additional","affiliation":[{"name":"Tencent Robotics X, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,12,5]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919887447"},{"key":"e_1_2_2_2_1","volume-title":"Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems 29","author":"Bellemare Marc","year":"2016","unstructured":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. 2016. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems 29 (2016)."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356536"},{"key":"e_1_2_2_4_1","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop Christopher M","unstructured":"Christopher M Bishop and Nasser M Nasrabadi. 2006. Pattern Recognition and Machine Learning. Vol. 4. Springer."},{"key":"e_1_2_2_5_1","volume-title":"Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349","author":"Bowman Samuel R","year":"2015","unstructured":"Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 (2015)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2485895.2485906"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480512"},{"key":"e_1_2_2_8_1","unstructured":"R\u00e9mi Coulom. 2005. Bayesian Elo Rating. https:\/\/www.remi-coulom.fr\/Bayesian-Elo\/"},{"key":"e_1_2_2_9_1","volume-title":"Creto Augusto Vidal, Joaquim B Cavalcante-Neto, Paul G Kry, and Victor B Zordan.","author":"da Silva Danilo Borges","year":"2017","unstructured":"Danilo Borges da Silva, Rubens Fernandes Nunes, Creto Augusto Vidal, Joaquim B Cavalcante-Neto, Paul G Kry, and Victor B Zordan. 2017. Tunable robustness: An artificial contact strategy with virtual actuator control for balance. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 499--510."},{"key":"e_1_2_2_10_1","volume-title":"Computer Graphics Forum","author":"Silva Marco Da","unstructured":"Marco Da Silva, Yeuhi Abe, and Jovan Popovi\u0107. 2008. Simulation of human motion data using short-horizon model-predictive control. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 371--380."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786784.2786802"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480527"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508399"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"e_1_2_2_15_1","volume-title":"Tstarbot-x: An open-sourced and comprehensive study for efficient league training in starcraft ii full game. arXiv preprint arXiv:2011.13729","author":"Han Lei","year":"2020","unstructured":"Lei Han, Jiechao Xiong, Peng Sun, Xinghai Sun, Meng Fang, Qingwei Guo, Qiaobo Chen, Tengfei Shi, Hongsheng Yu, Xipeng Wu, and Zhengyou Zhang. 2020. Tstarbot-x: An open-sourced and comprehensive study for efficient league training in starcraft ii full game. arXiv preprint arXiv:2011.13729 (2020)."},{"key":"e_1_2_2_16_1","volume-title":"Synthesizing Physical Character-Scene Interactions. arXiv preprint arXiv:2302.00883","author":"Hassan Mohamed","year":"2023","unstructured":"Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. 2023. Synthesizing Physical Character-Scene Interactions. arXiv preprint arXiv:2302.00883 (2023)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417836"},{"key":"e_1_2_2_18_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Higgins Irina","year":"2017","unstructured":"Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1833349.1778770"},{"key":"e_1_2_2_20_1","volume-title":"Generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS) 29","author":"Ho Jonathan","year":"2016","unstructured":"Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS) 29 (2016)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/218380.218414"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2019627.2019637"},{"key":"e_1_2_2_23_1","volume-title":"PADL: Language-Directed Physics-Based Character Control. In SIGGRAPH Asia 2022 Conference Papers. 1--9.","author":"Juravsky Jordan","year":"2022","unstructured":"Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. 2022. PADL: Language-Directed Physics-Based Character Control. In SIGGRAPH Asia 2022 Conference Papers. 1--9."},{"key":"e_1_2_2_24_1","volume-title":"Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 117--126","author":"Kim Manmyung","year":"2012","unstructured":"Manmyung Kim, Youngseok Hwang, Kyunglyul Hyun, and Jehee Lee. 2012. Tiling motion patches. In Proceedings of the ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 117--126."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531385"},{"key":"e_1_2_2_26_1","volume-title":"Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114","author":"Kingma Diederik P","year":"2013","unstructured":"Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2008.22"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983616"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1028523.1028535"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1179352.1141972"},{"key":"e_1_2_2_31_1","volume-title":"Deep Compliant Control. In ACM SIGGRAPH 2022 Conference Proceedings. 1--9.","author":"Lee Seunghwan","year":"2022","unstructured":"Seunghwan Lee, Phil Sik Chang, and Jehee Lee. 2022. Deep Compliant Control. In ACM SIGGRAPH 2022 Conference Proceedings. 1--9."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459774"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1833349.1781155"},{"key":"e_1_2_2_34_1","first-page":"1","article-title":"Ganimator: Neural motion synthesis from a single sequence","volume":"41","author":"Li Peizhuo","year":"2022","unstructured":"Peizhuo Li, Kfir Aberman, Zihan Zhang, Rana Hanocka, and Olga Sorkine-Hornung. 2022. Ganimator: Neural motion synthesis from a single sequence. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--12.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392422"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/1218064.1218093"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201315"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982424"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1833349.1778865"},{"key":"e_1_2_2_40_1","volume-title":"Daniel Hennes, Wojciech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al.","author":"Liu Siqi","year":"2022","unstructured":"Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Ali Eslami, Daniel Hennes, Wojciech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. 2022. From motor control to team play in simulated humanoid football. Science Robotics 7, 69 (2022), eabo0235."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1576246.1531386"},{"key":"e_1_2_2_42_1","volume-title":"Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning.","author":"Makoviychuk Viktor","year":"2021","unstructured":"Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. 2021. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392474"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185539"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1576246.1531387"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1966394.1966395"},{"key":"e_1_2_2_47_1","volume-title":"International Conference on Machine Learning. PMLR, 2721--2730","author":"Ostrovski Georg","year":"2017","unstructured":"Georg Ostrovski, Marc G Bellemare, A\u00e4ron Oord, and R\u00e9mi Munos. 2017. Count-based exploration with neural density models. In International Conference on Machine Learning. PMLR, 2721--2730."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356501"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201311"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925881"},{"key":"e_1_2_2_51_1","volume-title":"Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems 32","author":"Peng Xue Bin","year":"2019","unstructured":"Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. Mcp: Learning composable hierarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems 32 (2019)."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530110"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459670"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/122718.122755"},{"key":"e_1_2_2_55_1","volume-title":"Aaron Van den Oord, and Oriol Vinyals","author":"Razavi Ali","year":"2019","unstructured":"Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse highfidelity images with vq-vae-2. Advances in Neural Information Processing Systems (NeurIPS) 32 (2019)."},{"key":"e_1_2_2_56_1","volume-title":"Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv:1805.11063","author":"Roy Aurko","year":"2018","unstructured":"Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. 2018. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv:1805.11063 (2018)."},{"key":"e_1_2_2_57_1","volume-title":"Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell.","author":"Rusu Andrei A","year":"2015","unstructured":"Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. 2015. Policy distillation. arXiv preprint arXiv:1511.06295 (2015)."},{"key":"e_1_2_2_58_1","volume-title":"Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347","author":"Schulman John","year":"2017","unstructured":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)."},{"key":"e_1_2_2_59_1","volume-title":"International Conference on Machine Learning (ICML). 8655--8664","author":"Shao Huajie","year":"2020","unstructured":"Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. 2020. Controlvae: Controllable variational autoencoder. In International Conference on Machine Learning (ICML). 8655--8664."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1409060.1409067"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1315184.1315194"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1342250.1342271"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2010.257"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1275808.1276511"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcss.2007.08.009"},{"key":"e_1_2_2_66_1","volume-title":"Tleague: A framework for competitive self-play based distributed multi-agent reinforcement learning. arXiv preprint arXiv:2011.12895","author":"Sun Peng","year":"2020","unstructured":"Peng Sun, Jiechao Xiong, Lei Han, Xinghai Sun, Shuxing Li, Jiawei Xu, Meng Fang, and Zhengyou Zhang. 2020. Tleague: A framework for competitive self-play based distributed multi-agent reinforcement learning. arXiv preprint arXiv:2011.12895 (2020)."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601121"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.6059"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588432.3591541"},{"key":"e_1_2_2_70_1","unstructured":"Aaron Van Den Oord Oriol Vinyals et al. 2017. Neural discrete representation learning. Advances in Neural Information Processing Systems (NeurIPS) 30 (2017)."},{"key":"e_1_2_2_71_1","doi-asserted-by":"crossref","unstructured":"Oriol Vinyals Igor Babuschkin Wojciech M Czarnecki Micha\u00ebl Mathieu Andrew Dudzik Junyoung Chung David H Choi Richard Powell Timo Ewalds Petko Georgiev et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575 7782 (2019) 350--354.","DOI":"10.1038\/s41586-019-1724-z"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/1805964.1805970"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392381"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459761"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530067"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356499"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661271"},{"key":"e_1_2_2_78_1","volume-title":"Nam Hee Kim, and Michiel van de Panne.","author":"Xie Zhaoming","year":"2020","unstructured":"Zhaoming Xie, Hung Yu Ling, Nam Hee Kim, and Michiel van de Panne. 2020. Allsteps: curriculum-driven learning of stepping stone skills. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 213--224."},{"key":"e_1_2_2_79_1","volume-title":"ACM SIGGRAPH 2022 Conference Proceedings. 1--9.","author":"Xie Zhaoming","unstructured":"Zhaoming Xie, Sebastian Starke, Hung Yu Ling, and Michiel van de Panne. 2022. Learning soccer juggling skills with layer-wise mixture-of-experts. In ACM SIGGRAPH 2022 Conference Proceedings. 1--9."},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550454.3555434"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/1399504.1360680"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276509"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459817"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201397"},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00589"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/545261.545276"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618397","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3618397","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T10:51:31Z","timestamp":1755773491000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3618397"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,5]]},"references-count":86,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,12,5]]}},"alternative-id":["10.1145\/3618397"],"URL":"https:\/\/doi.org\/10.1145\/3618397","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,5]]},"assertion":[{"value":"2023-12-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}