{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T04:25:57Z","timestamp":1790915157621,"version":"4.1.0"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,7,19]]},"abstract":"<jats:p>Real-time character control is an essential component for interactive experiences, with a broad range of applications, including physics simulations, video games, and virtual reality. The success of diffusion models for image synthesis has led to the use of these models for motion synthesis. However, the majority of these motion diffusion models are primarily designed for offline applications, where space-time models are used to synthesize an entire sequence of frames simultaneously with a pre-specified length. To enable real-time motion synthesis with diffusion model that allows time-varying controls, we propose A-MDM (Auto-regressive Motion Diffusion Model). Our conditional diffusion model takes an initial pose as input, and auto-regressively generates successive motion frames conditioned on the previous frame. Despite its streamlined network architecture, which uses simple MLPs, our framework is capable of generating diverse, long-horizon, and high-fidelity motion sequences. Furthermore, we introduce a suite of techniques for incorporating interactive controls into A-MDM, such as task-oriented sampling, in-painting, and hierarchical reinforcement learning (See Figure 1). These techniques enable a pre-trained A-MDM to be efficiently adapted for a variety of new downstream tasks. We conduct a comprehensive suite of experiments to demonstrate the effectiveness of A-MDM, and compare its performance against state-of-the-art auto-regressive methods.<\/jats:p>","DOI":"10.1145\/3658140","type":"journal-article","created":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T10:47:57Z","timestamp":1721386077000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Interactive Character Control with Auto-Regressive Motion Diffusion Models"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3360-6706","authenticated-orcid":false,"given":"Yi","family":"Shi","sequence":"first","affiliation":[{"name":"Simon Fraser University, Burnaby, Canada"},{"name":"Shanghai Aritificial Intelligence Laboratory, Burnaby, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0740-8548","authenticated-orcid":false,"given":"Jingbo","family":"Wang","sequence":"additional","affiliation":[{"name":"Shanghai AI Lab, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2546-0885","authenticated-orcid":false,"given":"Xuekun","family":"Jiang","sequence":"additional","affiliation":[{"name":"Shanghai AI Lab, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3916-0550","authenticated-orcid":false,"given":"Bingkun","family":"Lin","sequence":"additional","affiliation":[{"name":"Xmov, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0777-9232","authenticated-orcid":false,"given":"Bo","family":"Dai","sequence":"additional","affiliation":[{"name":"Shanghai AI Lab, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3677-5655","authenticated-orcid":false,"given":"Xue Bin","family":"Peng","sequence":"additional","affiliation":[{"name":"Simon Fraser University, Vancouver, Canada"},{"name":"NVIDIA, Vancouver, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,19]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00724"},{"key":"e_1_2_2_2_1","volume-title":"HP-GAN: Probabilistic 3D human motion prediction via GAN. CoRR","author":"Barsoum Emad","year":"2017","unstructured":"Emad Barsoum, John Kender, and Zicheng Liu. 2017. HP-GAN: Probabilistic 3D human motion prediction via GAN. CoRR (2017)."},{"key":"e_1_2_2_3_1","volume-title":"Advances in Neural Information Processing Systems 34 (NeurIPS","author":"Bredell Gustav","year":"2021","unstructured":"Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertunc Erdil, and Ender Konukoglu. 2021. Minimizing the Blur Error of Variational Autoencoders. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021)."},{"key":"e_1_2_2_4_1","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:arXiv:1606.01540"},{"key":"e_1_2_2_5_1","volume-title":"Vladislav Golyanik, and Christian Theobalt.","author":"Dabral Rishabh","year":"2023","unstructured":"Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt. 2023. MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis. In CVPR."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/2919332.2919834"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01239"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00509"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01118"},{"key":"e_1_2_2_11_1","volume-title":"Denoising diffusion probabilistic models. NIPS","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. NIPS (2020)."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073663"},{"key":"e_1_2_2_13_1","volume-title":"Optimization, and Planning in 3D Scenes. CVPR","author":"Huang Siyuan","year":"2023","unstructured":"Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. 2023. Diffusion-based Generation, Optimization, and Planning in 3D Scenes. CVPR (2023)."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00205"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2022.108894"},{"key":"e_1_2_2_16_1","volume-title":"Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114","author":"Kingma Diederik P","year":"2013","unstructured":"Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530157"},{"key":"e_1_2_2_18_1","volume-title":"Auto-conditioned recurrent networks for extended complex human motion synthesis. arXiv preprint arXiv:1707.05363","author":"Li Zimo","year":"2017","unstructured":"Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, Zeng Huang, and Hao Li. 2017. Auto-conditioned recurrent networks for extended complex human motion synthesis. arXiv preprint arXiv:1707.05363 (2017)."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392422"},{"key":"e_1_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Andreas Lugmayr Martin Danelljan Andres Romero Fisher Yu Radu Timofte and Luc Gool. 2022. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. (2022).","DOI":"10.1109\/CVPR52688.2022.01117"},{"key":"e_1_2_2_21_1","volume-title":"Pretrained Diffusion Models for Unified Human Motion Synthesis. arXiv","author":"Ma Jianxin","year":"2022","unstructured":"Jianxin Ma, Shuai Bai, and Chang Zhou. 2022. Pretrained Diffusion Models for Unified Human Motion Synthesis. arXiv (2022)."},{"key":"e_1_2_2_22_1","volume-title":"AMASS: Archive of motion capture as surface shapes. In ICCV.","author":"Mahmood Naureen","year":"2019","unstructured":"Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. 2019. AMASS: Archive of motion capture as surface shapes. In ICCV."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.497"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3522618"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2021.09.014"},{"key":"e_1_2_2_26_1","volume-title":"Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint arXiv:1805.06485","author":"Pavllo Dario","year":"2018","unstructured":"Dario Pavllo, David Grangier, and Michael Auli. 2018. Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint arXiv:1805.06485 (2018)."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201311"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530110"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459670"},{"key":"e_1_2_2_30_1","doi-asserted-by":"crossref","unstructured":"Mathis Petrovich Michael J Black and G\u00fcl Varol. 2021. Action-conditioned 3D human motion synthesis with transformer VAE. In ICCV. 10985--10995.","DOI":"10.1109\/ICCV48922.2021.01080"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20047-2_28"},{"key":"e_1_2_2_32_1","volume-title":"Advances in Neural Information Processing Systems 34 (NeurIPS","author":"Prafulla Dhariwal Alexander Nichol","year":"2021","unstructured":"Alexander Nichol Prafulla Dhariwal. 2021. Diffusion Models Beat GANs on Image Synthesis. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021)."},{"key":"e_1_2_2_33_1","volume-title":"Single Motion Diffusion. arXiv","author":"Raab Sigal","year":"2023","unstructured":"Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. 2023. Single Motion Diffusion. arXiv (2023)."},{"key":"e_1_2_2_34_1","doi-asserted-by":"crossref","unstructured":"Davis Rempe Tolga Birdal Aaron Hertzmann Jimei Yang Srinath Sridhar and Leonidas J Guibas. 2021. Humor: 3d human motion model for robust pose estimation. In ICCV.","DOI":"10.1109\/ICCV48922.2021.01129"},{"key":"e_1_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Robin Rombach Andreas Blattmann Dominik Lorenz Patrick Esser and Bj\u00f6rn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models.","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_2_2_36_1","doi-asserted-by":"crossref","unstructured":"Nataniel Ruiz Yuanzhen Li Varun Jampani Yael Pritch Michael Rubinstein and Kfir Aberman. 2022. DreamBooth: Fine Tuning Text-to-image Diffusion Models for Subject-Driven Generation. (2022).","DOI":"10.1109\/CVPR52729.2023.02155"},{"key":"e_1_2_2_37_1","volume-title":"Burcu Karagol Ayan, Tim Salimans, et al.","author":"Saharia Chitwan","year":"2022","unstructured":"Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. NIPS (2022)."},{"key":"e_1_2_2_38_1","volume-title":"Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347","author":"Schulman John","year":"2017","unstructured":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)."},{"key":"e_1_2_2_39_1","volume-title":"Human Motion Diffusion as a Generative Prior. arXiv","author":"Shafir Yonatan","year":"2023","unstructured":"Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. 2023. Human Motion Diffusion as a Generative Prior. arXiv (2023)."},{"key":"e_1_2_2_40_1","volume-title":"Denoising diffusion implicit models. ICLR","author":"Song Jiaming","year":"2021","unstructured":"Jiaming Song, Chenlin Meng, and Stefano Ermon. 2021. Denoising diffusion implicit models. ICLR (2021)."},{"key":"e_1_2_2_41_1","volume-title":"arXiv preprint arXiv:2303.01469","author":"Song Yang","year":"2023","unstructured":"Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency Models. arXiv preprint arXiv:2303.01469 (2023)."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356505"},{"key":"e_1_2_2_43_1","volume-title":"CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. arXiv preprint arXiv:2305.02195","author":"Tessler Chen","year":"2023","unstructured":"Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. 2023. CALM: Conditional Adversarial Latent Models for Directable Virtual Characters. arXiv preprint arXiv:2305.02195 (2023)."},{"key":"e_1_2_2_44_1","volume-title":"Human motion diffusion model. ICLR","author":"Tevet Guy","year":"2023","unstructured":"Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2023. Human motion diffusion model. ICLR (2023)."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01981"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01203"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530067"},{"key":"e_1_2_2_48_1","doi-asserted-by":"crossref","unstructured":"Chen Xin Biao Jiang Wen Liu Zilong Huang Bin Fu Tao Chen Jingyi Yu and Gang Yu. 2023. Executing your Commands via Motion Diffusion in Latent Space. In CVPR.","DOI":"10.1109\/CVPR52729.2023.01726"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00449"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550454.3555434"},{"key":"e_1_2_2_51_1","volume-title":"MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model. arXiv","author":"Zhang Mingyuan","year":"2022","unstructured":"Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022. MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model. arXiv (2022)."},{"key":"e_1_2_2_52_1","volume-title":"ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. arXiv","author":"Zhang Mingyuan","year":"2023","unstructured":"Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. 2023a. ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. arXiv (2023)."},{"key":"e_1_2_2_53_1","doi-asserted-by":"crossref","unstructured":"Zihan Zhang Richard Liu Kfir Aberman and Rana Hanocka. 2023b. TEDi: Temporally-Entangled Diffusion for Long-Term Motion Synthesis. arXiv:2307.15042 [cs.CV]","DOI":"10.1145\/3641519.3657515"},{"key":"e_1_2_2_54_1","volume-title":"EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Motion Generation. arXiv preprint arXiv:2312.02256","author":"Zhou Wenyang","year":"2023","unstructured":"Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. 2023. EMDM: Efficient Motion Diffusion Model for Fast, High-Quality Motion Generation. arXiv preprint arXiv:2312.02256 (2023)."},{"key":"e_1_2_2_55_1","volume-title":"On the Continuity of Rotation Representations in Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Zhou Yi","year":"2019","unstructured":"Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. 2019. On the Continuity of Rotation Representations in Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658140","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3658140","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:05:54Z","timestamp":1750277154000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658140"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,19]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7,19]]}},"alternative-id":["10.1145\/3658140"],"URL":"https:\/\/doi.org\/10.1145\/3658140","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,19]]},"assertion":[{"value":"2024-07-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}