{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T02:13:00Z","timestamp":1774059180664,"version":"3.50.1"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,7,1]],"date-time":"2022-07-01T00:00:00Z","timestamp":1656633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62102403, No. 61872440 and No. 62061136007"],"award-info":[{"award-number":["No. 62102403, No. 61872440 and No. 62061136007"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Royal Society Newton Advanced Fellowship","award":["No. NAF\\R2\\192151"],"award-info":[{"award-number":["No. NAF\\R2\\192151"]}]},{"name":"the Beijing Municipal Natural Science Foundation for Distinguished Young Scholars","award":["No. JQ21013"],"award-info":[{"award-number":["No. JQ21013"]}]},{"name":"the Youth Innovation Promotion Association CAS"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>Sketches, which are simple and concise, have been used in recent deep image synthesis methods to allow intuitive generation and editing of facial images. However, it is nontrivial to extend such methods to video editing due to various challenges, ranging from appropriate manipulation propagation and fusion of multiple editing operations to ensure temporal coherence and visual quality. To address these issues, we propose a novel sketch-based facial video editing framework, in which we represent editing manipulations in latent space and propose specific propagation and fusion modules to generate high-quality video editing results based on StyleGAN3. Specifically, we first design an optimization approach to represent sketch editing manipulations by editing vectors, which are propagated to the whole video sequence using a proper strategy to cope with different editing needs. Specifically, input editing operations are classified into two categories: temporally consistent editing and temporally variant editing. The former (e.g., change of face shape) is applied to the whole video sequence directly, while the latter (e.g., change of facial expression or dynamics) is propagated with the guidance of expression or only affects adjacent frames in a given time window. Since users often perform different editing operations in multiple frames, we further present a region-aware fusion approach to fuse diverse editing effects. Our method supports video editing on facial structure and expression movement by sketch, which cannot be achieved by previous works. Both qualitative and quantitative evaluations show the superior editing ability of our system to existing and alternative solutions.<\/jats:p>","DOI":"10.1145\/3528223.3530056","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T21:06:27Z","timestamp":1658523987000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["DeepFaceVideoEditing"],"prefix":"10.1145","volume":"41","author":[{"given":"Feng-Lin","family":"Liu","sequence":"first","affiliation":[{"name":"CAS and University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shu-Yu","family":"Chen","sequence":"additional","affiliation":[{"name":"CAS and University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yu-Kun","family":"Lai","sequence":"additional","affiliation":[{"name":"Cardiff University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunpeng","family":"Li","sequence":"additional","affiliation":[{"name":"CAS and University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yue-Ren","family":"Jiang","sequence":"additional","affiliation":[{"name":"CAS and University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hongbo","family":"Fu","sequence":"additional","affiliation":[{"name":"City University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lin","family":"Gao","sequence":"additional","affiliation":[{"name":"CAS and University of Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,22]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447648"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459805"},{"key":"e_1_2_2_3_1","volume-title":"Bermano","author":"Alaluf Yuval","year":"2021","unstructured":"Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit H. Bermano. 2021b. HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing. CoRR abs\/2111.15666 (2021)."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015706.1015729"},{"key":"e_1_2_2_5_1","unstructured":"Mikolaj Binkowski Danica J. Sutherland Michael Arbel and Arthur Gretton. 2018. Demystifying MMD GANs. In ICLR."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459760"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392386"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2019.00038"},{"key":"e_1_2_2_9_1","doi-asserted-by":"crossref","unstructured":"Leon A. Gatys Alexander S. Ecker and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In CVPR. 2414--2423.","DOI":"10.1109\/CVPR.2016.265"},{"key":"e_1_2_2_10_1","volume-title":"Generative Adversarial Networks. CoRR abs\/1406.2661","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Networks. CoRR abs\/1406.2661 (2014). arXiv:1406.2661"},{"key":"e_1_2_2_11_1","unstructured":"Shuyang Gu Jianmin Bao Hao Yang Dong Chen Fang Wen and Lu Yuan. 2019. Mask-Guided Portrait Editing With Conditional GANs. In CVPR. 3436--3445."},{"key":"e_1_2_2_12_1","volume-title":"Ganspace: Discovering interpretable gan controls.","author":"H\u00e4rk\u00f6nen Erik","year":"2020","unstructured":"Erik H\u00e4rk\u00f6nen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. 2020. Ganspace: Discovering interpretable gan controls. (2020), 9841--9850."},{"key":"e_1_2_2_13_1","unstructured":"Martin Heusel Hubert Ramsauer Thomas Unterthiner Bernhard Nessler and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In NeurIPS. 6626--6637."},{"key":"e_1_2_2_14_1","volume-title":"Jittor: a novel deep learning framework with meta-operators and unified graph execution. Science China Information Sciences 63, 222103","author":"Hu Shi-Min","year":"2020","unstructured":"Shi-Min Hu, Dun Liang, Guo-Ye Yang, Guo-Wei Yang, and Wen-Yang Zhou. 2020. Jittor: a novel deep learning framework with meta-operators and unified graph execution. Science China Information Sciences 63, 222103 (2020), 1--21."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982398"},{"key":"e_1_2_2_16_1","volume-title":"Efros","author":"Isola Phillip","year":"2017","unstructured":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR. 5967--5976."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323006"},{"key":"e_1_2_2_18_1","doi-asserted-by":"crossref","unstructured":"Youngjoo Jo and Jongyoul Park. 2019. SC-FEGAN: Face Editing Generative Adversarial Network With User's Sketch and Color. In ICCV. 1745--1753.","DOI":"10.1109\/ICCV.2019.00183"},{"key":"e_1_2_2_19_1","volume-title":"Alias-free generative adversarial networks. NeurIPS","author":"Karras Tero","year":"2021","unstructured":"Tero Karras, Miika Aittala, Samuli Laine, Erik H\u00e4rk\u00f6nen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021. Alias-free generative adversarial networks. NeurIPS (2021), 852--863."},{"key":"e_1_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Tero Karras Samuli Laine and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In CVPR. 4401--4410.","DOI":"10.1109\/CVPR.2019.00453"},{"key":"e_1_2_2_21_1","doi-asserted-by":"crossref","unstructured":"Tero Karras Samuli Laine Miika Aittala Janne Hellsten Jaakko Lehtinen and Timo Aila. 2020. Analyzing and Improving the Image Quality of StyleGAN. In CVPR. 8107--8116.","DOI":"10.1109\/CVPR42600.2020.00813"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480546"},{"key":"e_1_2_2_23_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_24_1","doi-asserted-by":"crossref","unstructured":"Cheng-Han Lee Ziwei Liu Lingyun Wu and Ping Luo. 2020. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In CVPR. 5548--5557.","DOI":"10.1109\/CVPR42600.2020.00559"},{"key":"e_1_2_2_25_1","unstructured":"Chenyang Lei and Qifeng Chen. 2019. Fully automatic video colorization with self-regularization and diversity. In CVPR. 3753--3761."},{"key":"e_1_2_2_26_1","volume-title":"Antonio Torralba, and Sanja Fidler.","author":"Ling Huan","year":"2021","unstructured":"Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, and Sanja Fidler. 2021. EditGAN: High-Precision Semantic Image Editing. NeurIPS (2021), 16331--16345."},{"key":"e_1_2_2_27_1","first-page":"213","article-title":"Image Generation from Sketch Constraint Using Contextual GAN","volume":"11220","author":"Lu Yongyi","year":"2018","unstructured":"Yongyi Lu, Shangzhe Wu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Image Generation from Sketch Constraint Using Contextual GAN. In ECCV, Vol. 11220. 213--228.","journal-title":"ECCV"},{"key":"e_1_2_2_28_1","article-title":"PhotoApp: Photorealistic appearance editing of head portraits","volume":"40","author":"Mallikarjun BR","year":"2021","unstructured":"BR Mallikarjun, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed A Elgharib, et al. 2021. PhotoApp: Photorealistic appearance editing of head portraits. ACM Trans. Graph. 40, 4 (2021), 44:1--44:16.","journal-title":"ACM Trans. Graph."},{"key":"e_1_2_2_29_1","volume-title":"Deep Video Color Propagation. In British Machine Vision Conference (BMVC). 128","author":"Meyer Simone","unstructured":"Simone Meyer, Victor Cornill\u00e8re, Abdelaziz Djelouah, Christopher Schroers, and Markus H. Gross. 2018. Deep Video Color Propagation. In British Machine Vision Conference (BMVC). 128."},{"key":"e_1_2_2_30_1","volume-title":"Conditional Generative Adversarial Nets. CoRR abs\/1411.1784","author":"Mirza Mehdi","year":"2014","unstructured":"Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. CoRR abs\/1411.1784 (2014). arXiv:1411.1784"},{"key":"e_1_2_2_31_1","doi-asserted-by":"crossref","unstructured":"Prashant Pandey Mrigank Raman Sumanth Varambally and Prathosh AP. 2021. Generalization on Unseen Domains via Inference-Time Label-Preserving Target Projections. In CVPR. 12924--12933.","DOI":"10.1109\/CVPR46437.2021.01273"},{"key":"e_1_2_2_32_1","unstructured":"A. Paszke S. Gross F. Massa A. Lerer J. Bradbury G. Chanan T. Killeen Z. Lin N. Gimelshein L. Antiga and et al. 2019. PyTorch: An Imperative Style High-Performance Deep Learning Library. In NeurIPS. 8024--8035."},{"key":"e_1_2_2_33_1","volume-title":"Styleclip: Text-driven manipulation of stylegan imagery. In ICCV. 2085--2094.","author":"Patashnik Or","year":"2021","unstructured":"Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV. 2085--2094."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201393"},{"key":"e_1_2_2_35_1","volume-title":"Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML, Marina Meila and Tong Zhang (Eds.), Vol. 139. 8748--8763."},{"key":"e_1_2_2_36_1","doi-asserted-by":"crossref","unstructured":"Elad Richardson Yuval Alaluf Or Patashnik Yotam Nitzan Yaniv Azar Stav Shapiro and Daniel Cohen-Or. 2021. Encoding in Style: A StyleGAN Encoder for Image-to-Image Translation. In CVPR. 2287--2296.","DOI":"10.1109\/CVPR46437.2021.00232"},{"key":"e_1_2_2_37_1","volume-title":"Pivotal Tuning for Latent-based Editing of Real Images. CoRR abs\/2106.05744","author":"Roich Daniel","year":"2021","unstructured":"Daniel Roich, Ron Mokady, Amit H. Bermano, and Daniel Cohen-Or. 2021. Pivotal Tuning for Latent-based Editing of Real Images. CoRR abs\/2106.05744 (2021)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-018-1089-z"},{"key":"e_1_2_2_39_1","doi-asserted-by":"crossref","unstructured":"Yujun Shen Jinjin Gu Xiaoou Tang and Bolei Zhou. 2020. Interpreting the latent space of gans for semantic face editing. In CVPR. 9243--9252.","DOI":"10.1109\/CVPR42600.2020.00926"},{"key":"e_1_2_2_40_1","volume-title":"First order motion model for image animation. NeurIPS","author":"Siarohin Aliaksandr","year":"2019","unstructured":"Aliaksandr Siarohin, St\u00e9phane Lathuili\u00e8re, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. First order motion model for image animation. NeurIPS (2019), 7137--7147."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417803"},{"key":"e_1_2_2_42_1","doi-asserted-by":"crossref","unstructured":"Ayush Tewari Mohamed Elgharib Gaurav Bharaj Florian Bernard Hans-Peter Seidel Patrick P\u00e9rez Michael Zollhofer and Christian Theobalt. 2020b. StyleRig: Rigging stylegan for 3D control over portrait images. In CVPR. 6142--6151.","DOI":"10.1109\/CVPR42600.2020.00618"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392453"},{"key":"e_1_2_2_44_1","unstructured":"Yu Tian Jian Ren Menglei Chai Kyle Olszewski Xi Peng Dimitris N. Metaxas and Sergey Tulyakov. 2021. A Good Image Generator Is What You Need for High-Resolution Video Synthesis. In ICLR."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459838"},{"key":"e_1_2_2_46_1","volume-title":"Tracking Emerges by Colorizing Videos","author":"Vondrick Carl","unstructured":"Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. 2018. Tracking Emerges by Colorizing Videos. In ECCV, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.), Vol. 11217. 402--419."},{"key":"e_1_2_2_47_1","unstructured":"Ting-Chun Wang Ming-Yu Liu Andrew Tao Guilin Liu Bryan Catanzaro and Jan Kautz. 2019. Few-shot Video-to-Video Synthesis. In NeurIPS. 5014--5025."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459814"},{"key":"e_1_2_2_49_1","doi-asserted-by":"crossref","unstructured":"Chao Yang Xin Lu Zhe Lin Eli Shechtman Oliver Wang and Hao Li. 2017. High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis. In CVPR. 4076--4084.","DOI":"10.1109\/CVPR.2017.434"},{"key":"e_1_2_2_50_1","volume-title":"Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches","author":"Yang Shuai","unstructured":"Shuai Yang, Zhangyang Wang, Jiaying Liu, and Zongming Guo. 2020. Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches. In ECCV, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.), Vol. 12360. 601--617."},{"key":"e_1_2_2_51_1","doi-asserted-by":"crossref","unstructured":"Xu Yao Alasdair Newson Yann Gousseau and Pierre Hellier. 2021. A Latent Transformer for Disentangled Face Editing in Images and Videos. In ICCV. 13789--13798.","DOI":"10.1109\/ICCV48922.2021.01353"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-021-01515-2"},{"key":"e_1_2_2_53_1","doi-asserted-by":"crossref","unstructured":"Bo Zhang Mingming He Jing Liao Pedro V. Sander Lu Yuan Amine Bermak and Dong Chen. 2019. Deep Exemplar-Based Video Colorization. In CVPR. 8052--8061.","DOI":"10.1109\/CVPR.2019.00824"},{"key":"e_1_2_2_54_1","doi-asserted-by":"crossref","unstructured":"Richard Zhang Phillip Isola Alexei A Efros Eli Shechtman and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR. 586--595.","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480537"},{"key":"e_1_2_2_56_1","volume-title":"SEAN: Image Synthesis With Semantic Region-Adaptive Normalization. In CVPR. 5103--5112.","author":"Zhu Peihao","year":"2020","unstructured":"Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. 2020. SEAN: Image Synthesis With Semantic Region-Adaptive Normalization. In CVPR. 5103--5112."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530056","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3528223.3530056","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:25Z","timestamp":1750186945000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530056"}},"subtitle":["sketch-based deep editing of face videos"],"short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":56,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,7]]}},"alternative-id":["10.1145\/3528223.3530056"],"URL":"https:\/\/doi.org\/10.1145\/3528223.3530056","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7]]},"assertion":[{"value":"2022-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}