{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T20:40:56Z","timestamp":1784148056409,"version":"3.55.0"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2025,8,1]]},"abstract":"<jats:p>\n                    Storyboarding is widely used for creating 3D animations. Animators use the 2D sketches in storyboards as references to craft the desired 3D animations through a trial-and-error process. The traditional approach requires exceptional expertise and is both labor-intensive and time-consuming. Consequently, there is a high demand for automated methods that can directly translate 2D storyboard sketches into 3D animations. This task is under-explored to date and inspired by the significant advancements of motion diffusion models, we propose to address it from the perspective of conditional motion synthesis. We thus present\n                    <jats:italic toggle=\"yes\">Sketch2Anim<\/jats:italic>\n                    , composed of two key modules for sketch constraint understanding and motion generation. Specifically, due to the large domain gap between the 2D sketch and 3D motion, instead of directly conditioning on 2D inputs, we design a 3D conditional motion generator that simultaneously leverages 3D keyposes, joint trajectories, and action words, to achieve precise and fine-grained motion control. Then, we invent a neural mapper dedicated to aligning user-provided 2D sketches with their corresponding 3D keyposes and trajectories in a shared embedding space, enabling,\n                    <jats:italic toggle=\"yes\">for the first time<\/jats:italic>\n                    , direct 2D control of motion generation. Our approach successfully transfers storyboards into high-quality 3D motions and inherently supports direct 3D animation editing, thanks to the flexibility of our multi-conditional motion generator. Comprehensive experiments and evaluations, and a user perceptual study demonstrate the effectiveness of our approach.\n                  <\/jats:p>\n                  <jats:p>The code, data, trained models, and sketch-based motion designing interface are at https:\/\/zhongleilz.github.io\/Sketch2Anim\/.<\/jats:p>","DOI":"10.1145\/3731167","type":"journal-article","created":{"date-parts":[[2025,7,27]],"date-time":"2025-07-27T04:02:22Z","timestamp":1753588942000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation"],"prefix":"10.1145","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1778-9282","authenticated-orcid":false,"given":"Lei","family":"Zhong","sequence":"first","affiliation":[{"name":"University of Edinburgh, Edinburgh, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4539-0634","authenticated-orcid":false,"given":"Chuan","family":"Guo","sequence":"additional","affiliation":[{"name":"Snap Inc., New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2408-403X","authenticated-orcid":false,"given":"Yiming","family":"Xie","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7343-8066","authenticated-orcid":false,"given":"Jiawei","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Edinburgh, Edinburgh, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0448-4957","authenticated-orcid":false,"given":"Changjian","family":"Li","sequence":"additional","affiliation":[{"name":"University of Edinburgh, Edinburgh, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,7,27]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"SKEL-Betweener: a Neural Motion Rig for Interactive Motion Authoring. ACM Transactions on Graphics (TOG)","author":"Agrawal Dhruv","year":"2024","unstructured":"Dhruv Agrawal, Jakob Buhmann, Dominik Borer, Robert W Sumner, and Martin Guay. 2024. SKEL-Betweener: a Neural Motion Rig for Interactive Motion Authoring. ACM Transactions on Graphics (TOG) (2024)."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3680528.3687559"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.00051"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19815-1_11"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980240"},{"key":"e_1_2_2_6_1","volume-title":"MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls. arXiv preprint arXiv:2407.21136","author":"Bian Yuxuan","year":"2024","unstructured":"Yuxuan Bian, Ailing Zeng, Xuan Ju, Xian Liu, Zhaoyang Zhang, Wei Liu, and Qiang Xu. 2024. MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls. arXiv preprint arXiv:2407.21136 (2024)."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530106"},{"key":"e_1_2_2_8_1","volume-title":"Motion Matching - The Road to Next Gen Animation. https:\/\/www.youtube.com\/watch?v=z_wpgHFSWss&t=658s. Presented at Nucl.ai","author":"B\u00fcttner Michael","year":"2015","unstructured":"Michael B\u00fcttner and Simon Clavet. 2015. Motion Matching - The Road to Next Gen Animation. https:\/\/www.youtube.com\/watch?v=z_wpgHFSWss&t=658s. Presented at Nucl.ai 2015."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01726"},{"key":"e_1_2_2_10_1","volume-title":"John P Lewis, Yeongho Seol, Seokpyo Hong, Haegwang Eom, Sunjin Jung, and Junyong Noh.","author":"Choi Byungkuk","year":"2016","unstructured":"Byungkuk Choi, Roger Blanco i Ribera, John P Lewis, Yeongho Seol, Seokpyo Hong, Haegwang Eom, Sunjin Jung, and Junyong Noh. 2016. SketchiMo: sketch-based motion editing for articulated characters. ACM Transactions on Graphics (ToG) (2016)."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03198.x"},{"key":"e_1_2_2_12_1","volume-title":"ACM SIGGRAPH 2024 Conference Papers.","author":"Cohan Setareh","unstructured":"Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. 2024. Flexible motion in-betweening with diffusion models. In ACM SIGGRAPH 2024 Conference Papers."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-72640-8_22"},{"key":"e_1_2_2_14_1","doi-asserted-by":"crossref","unstructured":"James Davis Maneesh Agrawala Erika Chuang Zoran Popovi\u0107 and David Salesin. 2006. A sketching interface for articulated figure animation. In Acm siggraph 2006 courses. 15\u2013es.","DOI":"10.1145\/1185657.1185776"},{"key":"e_1_2_2_15_1","volume-title":"Diffusion models beat gans on image synthesis. Advances in neural information processing systems","author":"Dhariwal Prafulla","year":"2021","unstructured":"Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems (2021)."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.01880"},{"key":"e_1_2_2_17_1","volume-title":"Monster mash: a single-view approach to casual 3D modeling and animation. ACM Transactions on Graphics (ToG)","author":"Dvoro\u017e\u0148\u00e5k Marek","year":"2020","unstructured":"Marek Dvoro\u017e\u0148\u00e5k, Daniel S\u1ef3kora, Cassidy Curtis, Brian Curless, Olga Sorkine-Hornung, and David Salesin. 2020. Monster mash: a single-view approach to casual 3D modeling and animation. ACM Transactions on Graphics (ToG) (2020)."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00143"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3641519.3657447"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508397"},{"key":"e_1_2_2_21_1","volume-title":"Space-time sketching of character animation. ACM Transactions on Graphics (ToG)","author":"Guay Martin","year":"2015","unstructured":"Martin Guay, R\u00e9mi Ronfard, Michael Gleicher, and Marie-Paule Cani. 2015. Space-time sketching of character animation. ACM Transactions on Graphics (ToG) (2015)."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.00186"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00509"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19833-5_34"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786784.2786785"},{"key":"e_1_2_2_26_1","volume-title":"Robust motion in-betweening. ACM Transactions on Graphics (TOG)","author":"Harvey F\u00e9lix G","year":"2020","unstructured":"F\u00e9lix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. 2020. Robust motion in-betweening. ACM Transactions on Graphics (TOG) (2020)."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1230100.1230123"},{"key":"e_1_2_2_28_1","volume-title":"Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598","author":"Ho Jonathan","year":"2022","unstructured":"Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/288392.288599"},{"key":"e_1_2_2_30_1","volume-title":"Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems","author":"Jiang Biao","year":"2023","unstructured":"Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems (2023)."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00205"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3596711.3596788"},{"key":"e_1_2_2_33_1","doi-asserted-by":"crossref","unstructured":"John Lasseter. 1998. Principles of traditional animation applied to 3D computer animation. In Seminal graphics: pioneering efforts that shaped the field.","DOI":"10.1145\/280811.281005"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-72940-9_4"},{"key":"e_1_2_2_35_1","volume-title":"Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG)","author":"Li Jiaman","year":"2023","unstructured":"Jiaman Li, Jiajun Wu, and C Karen Liu. 2023. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) (2023)."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530157"},{"key":"e_1_2_2_37_1","volume-title":"Yang","author":"Li Zhe","year":"2024","unstructured":"Zhe Li, Weihao Yuan, Yisheng He, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu, Weichao Shen, Yuan Dong, Zilong Dong, and Laurence T. Yang. 2024. LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning. In arXiv 2410.07093."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1923363.1923383"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3596711.3596800"},{"key":"e_1_2_2_40_1","volume-title":"Decoupled weight decay regularization. ArXiv","author":"Loshchilov Ilya","year":"2017","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. ArXiv (2017)."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i5.28226"},{"key":"e_1_2_2_42_1","unstructured":"Nkeeline. 2024. Keemap-Blender Rig Retargeting Addon. https:\/\/github.com\/nkeeline\/Keemap-Blender-Rig-ReTargeting-Addon."},{"key":"e_1_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Yichen Peng Zhengyu Huang Chunqi Zhao Haoran Xie Tsukasa Fukusato and Kazunori Miyata. 2021. Sketch-based human motion retrieval via shadow guidance. In 2021 Nicograph International (NicoInt).","DOI":"10.1109\/NICOINT52941.2021.00015"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00870"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.00153"},{"key":"e_1_2_2_46_1","volume-title":"Caiming Xiong, Silvio Savarese, et al.","author":"Qin Can","year":"2023","unstructured":"Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. 2023. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305.11147 (2023)."},{"key":"e_1_2_2_47_1","volume-title":"International conference on machine learning. PMLR, 8748\u20138763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748\u20138763."},{"key":"e_1_2_2_48_1","volume-title":"Proceedings of the 2006 ACM SIGGRAPH\/Eurographics symposium on Computer animation.","author":"Shin Hyun Joon","year":"2006","unstructured":"Hyun Joon Shin and Hyun Seok Oh. 2006. Fat graphs: constructing an interactive character with continuous controls. In Proceedings of the 2006 ACM SIGGRAPH\/Eurographics symposium on Computer animation."},{"key":"e_1_2_2_49_1","volume-title":"A method for animating children's drawings of the human figure. ACM Transactions on Graphics","author":"Smith Harrison Jesse","year":"2023","unstructured":"Harrison Jesse Smith, Qingyuan Zheng, Yifei Li, Somya Jain, and Jessica K Hodgins. 2023. A method for animating children's drawings of the human figure. ACM Transactions on Graphics (2023)."},{"key":"e_1_2_2_50_1","volume-title":"Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502","author":"Song Jiaming","year":"2020","unstructured":"Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)."},{"key":"e_1_2_2_51_1","volume-title":"International Conference on Machine Learning. PMLR.","author":"Song Jiaming","year":"2023","unstructured":"Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming-Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. 2023. Loss-guided diffusion models for plug-and-play controllable generation. In International Conference on Machine Learning. PMLR."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3606921"},{"key":"e_1_2_2_53_1","volume-title":"Human Motion Diffusion Model. In The Eleventh International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=SJ1kSyO2jwu","author":"Tevet Guy","year":"2023","unstructured":"Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. 2023. Human Motion Diffusion Model. In The Eleventh International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=SJ1kSyO2jwu"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1186562.1015740"},{"key":"e_1_2_2_55_1","volume-title":"Tlcontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135","author":"Wan Weilin","year":"2023","unstructured":"Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. 2023. Tlcontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135 (2023)."},{"key":"e_1_2_2_56_1","volume-title":"InterControl: Generate Human Motion Interactions by Controlling Every Joint. arXiv preprint arXiv:2311.15864","author":"Wang Zhenzhi","year":"2023","unstructured":"Zhenzhi Wang, Jingbo Wang, Dahua Lin, and Bo Dai. 2023. InterControl: Generate Human Motion Interactions by Controlling Every Joint. arXiv preprint arXiv:2311.15864 (2023)."},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00606"},{"key":"e_1_2_2_58_1","volume-title":"The animator's survival kit: a manual of methods, principles and formulas for classical, computer, games, stop motion and internet animators","author":"Williams Richard","unstructured":"Richard Williams. 2012. The animator's survival kit: a manual of methods, principles and formulas for classical, computer, games, stop motion and internet animators. Macmillan."},{"key":"e_1_2_2_59_1","volume-title":"Doodle Your Motion: Sketch-Guided Human Motion Generation","author":"Wu Zizhao","year":"2024","unstructured":"Zizhao Wu, Qin Wang, Xinyang Zheng, Jianglei Ye, Ping Yang, Yunhai Wang, and Yigang Wang. 2024. Doodle Your Motion: Sketch-Guided Human Motion Generation. IEEE Transactions on Visualization and Computer Graphics (2024)."},{"key":"e_1_2_2_60_1","volume-title":"The Twelfth International Conference on Learning Representations.","author":"Xie Yiming","year":"2024","unstructured":"Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. 2024. OmniControl: Control Any Joint at Any Time for Human Motion Generation. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20086-1_18"},{"key":"e_1_2_2_62_1","volume-title":"Xue Bin Peng, and Davis Rempe","author":"Yi Hongwei","year":"2024","unstructured":"Hongwei Yi, Justus Thies, Michael J. Black, Xue Bin Peng, and Davis Rempe. 2024. Generating Human Interaction Motions in Scenes with Text Control. arXiv:2404.10685 (2024)."},{"key":"e_1_2_2_63_1","volume-title":"Sketching human character animations by composing sequences from large motion database. The Visual Computer","author":"Yoo Innfarn","year":"2014","unstructured":"Innfarn Yoo, Juraj Vanek, Maria Nizovtseva, Nicoletta Adamo-Villani, and Bedrich Benes. 2014. Sketching human character animations by composing sequences from large motion database. The Visual Computer (2014)."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01415"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00355"},{"key":"e_1_2_2_66_1","volume-title":"MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model. arXiv preprint arXiv:2208.15001","author":"Zhang Mingyuan","year":"2022","unstructured":"Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022. MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model. arXiv preprint arXiv:2208.15001 (2022)."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i7.28567"},{"key":"e_1_2_2_68_1","volume-title":"Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems","author":"Zhao Shihao","year":"2024","unstructured":"Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong. 2024. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems (2024)."},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-73232-4_23"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3680528.3687593"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-72627-9_2"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-96530-3"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3731167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T17:54:17Z","timestamp":1774634057000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3731167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,27]]},"references-count":72,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,8,1]]}},"alternative-id":["10.1145\/3731167"],"URL":"https:\/\/doi.org\/10.1145\/3731167","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,27]]},"assertion":[{"value":"2025-07-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}