{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T17:46:55Z","timestamp":1785865615065,"version":"3.56.0"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,7,19]]},"abstract":"<jats:p>\n            We present an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a \"latent transparency\" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground\/background-conditioned layer generation, joint layer generation, structural control of layer contents,\n            <jats:italic>etc.<\/jats:italic>\n            A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.\n          <\/jats:p>","DOI":"10.1145\/3658150","type":"journal-article","created":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T14:47:57Z","timestamp":1721400477000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":53,"title":["Transparent Image Layer Diffusion using Latent Transparency"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3503-5791","authenticated-orcid":false,"given":"Lvmin","family":"Zhang","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8996-7327","authenticated-orcid":false,"given":"Maneesh","family":"Agrawala","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,19]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.32"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2907940"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3002176"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201275"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01767"},{"key":"e_1_2_1_6_1","unstructured":"Yogesh Balaji Seungjun Nah Xun Huang Arash Vahdat Jiaming Song Karsten Kreis Miika Aittala Timo Aila Samuli Laine Bryan Catanzaro et al. 2022. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)."},{"key":"e_1_2_1_7_1","volume-title":"animagine-xl-3.0. huggingface","year":"2024","unstructured":"cagliostrolab. 2024. animagine-xl-3.0. huggingface (2024)."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.02062"},{"key":"e_1_2_1_9_1","unstructured":"Guowei Chen Yi Liu Jian Wang Juncai Peng Yuying Hao Lutao Chu Shiyu Tang Zewu Wu Zeyu Chen Zhiliang Yu Yuning Du Qingqing Dang Xiaoguang Hu and Dianhai Yu. 2022. PP-Matting: High-Accuracy Natural Image Matting."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2023.3324591"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00842"},{"key":"e_1_2_1_12_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Couairon Guillaume","year":"2023","unstructured":"Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. 2023. DiffEdit: Diffusion-based semantic image editing with mask guidance. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_1_13_1","volume-title":"stable-diffusion-xl-1.0-inpainting-0.1. diffusers","year":"2024","unstructured":"diffusers. 2024. stable-diffusion-xl-1.0-inpainting-0.1. diffusers (2024)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592128"},{"key":"e_1_2_1_15_1","volume-title":"An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618","author":"Gal Rinon","year":"2022","unstructured":"Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022)."},{"key":"e_1_2_1_16_1","volume-title":"3rd International Conference on Learning Representations, ICLR","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). arXiv:1412.6572"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5917--5926","author":"Andrade Guerreiro Julian Jorge","year":"2023","unstructured":"Julian Jorge Andrade Guerreiro, Mitsuru Nakazawa, and Bj\u00f6rn Stenger. 2023. PCT-Net: Full Resolution Image Harmonization Using Pixel-Wise Color Transformations. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5917--5926."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01610"},{"key":"e_1_2_1_19_1","volume-title":"Prompt-to-Prompt Image Editing with Cross-Attention Control. In International Conference on Learning Representations (ICLR).","author":"Hertz Amir","year":"2023","unstructured":"Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023. Prompt-to-Prompt Image Editing with Cross-Attention Control. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_1_20_1","unstructured":"Jonathan Ho Ajay Jain and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. NeurIPS."},{"key":"e_1_2_1_21_1","volume-title":"LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685","author":"Hu Edward J","year":"2021","unstructured":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685 (2021)."},{"key":"e_1_2_1_22_1","unstructured":"Gabriel Ilharco Mitchell Wortsman Ross Wightman Cade Gordon Nicholas Carlini Rohan Taori Achal Dave Vaishaal Shankar Hongseok Namkoong John Miller Hannaneh Hajishirzi Ali Farhadi and Ludwig Schmidt. 2021. OpenCLIP."},{"key":"e_1_2_1_23_1","volume-title":"Imagic: Text-Based Real Image Editing with Diffusion Models. In Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Kawar Bahjat","year":"2023","unstructured":"Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023. Imagic: Text-Based Real Image Editing with Diffusion Models. In Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00246"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00246"},{"key":"e_1_2_1_26_1","volume-title":"arXiv:2304.02643","author":"Kirillov Alexander","year":"2023","unstructured":"Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll\u00e1r, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 (2023)."},{"key":"e_1_2_1_27_1","unstructured":"Yuval Kirstain Adam Polyak Uriel Singer Shahbuland Matiana Joe Penna and Omer Levy. 2023. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation."},{"key":"e_1_2_1_28_1","volume-title":"On fast sampling of diffusion probabilistic models. CoRR 2106","author":"Kong Zhifeng","year":"2021","unstructured":"Zhifeng Kong and Wei Ping. 2021. On fast sampling of diffusion probabilistic models. CoRR 2106 (2021)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13577"},{"key":"e_1_2_1_30_1","volume-title":"arXiv: 2306.05399","author":"Li Jiachen","year":"2023","unstructured":"Jiachen Li, Jitesh Jain, and Humphrey Shi. 2023b. Matting Anything. arXiv: 2306.05399 (2023)."},{"key":"e_1_2_1_31_1","unstructured":"Pengzhi Li QInxuan Huang Yikang Ding and Zhiheng Li. 2023a. LayerDiffusion: Layered Controlled Image Editing with Diffusion Models. arXiv:2305.18676 [cs.CV]"},{"key":"e_1_2_1_32_1","unstructured":"Haotian Liu Chunyuan Li Qingyang Wu and Yong Jae Lee. 2023. Visual Instruction Tuning. In NeurIPS."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00585"},{"key":"e_1_2_1_34_1","volume-title":"T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. arXiv preprint arXiv:2302.08453","author":"Mou Chong","year":"2023","unstructured":"Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 2023. T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. arXiv preprint arXiv:2302.08453 (2023)."},{"key":"e_1_2_1_35_1","volume-title":"Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741","author":"Nichol Alex","year":"2021","unstructured":"Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00688"},{"key":"e_1_2_1_37_1","volume-title":"SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. (July","author":"Podell Dustin","year":"2023","unstructured":"Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M\u00fcller, Joe Penna, and Robin Rombach. 2023. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. (July 2023). arXiv:2307.01952 [cs.CV]"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1910.10683"},{"key":"e_1_2_1_39_1","volume-title":"Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125","author":"Ramesh Aditya","year":"2022","unstructured":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_2_1_41_1","volume-title":"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242","author":"Ruiz Nataniel","year":"2022","unstructured":"Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2022. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 (2022)."},{"key":"e_1_2_1_42_1","volume-title":"Noise estimation for generative diffusion models. CoRR 2104","author":"San-Roman Robin","year":"2021","unstructured":"Robin San-Roman, Eliya Nachmani, and Lior Wolf. 2021. Noise estimation for generative diffusion models. CoRR 2104 (2021)."},{"key":"e_1_2_1_43_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Schuhmann Christoph","year":"2022","unstructured":"Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. LAION-5B: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_2_1_44_1","unstructured":"Christoph Schuhmann and Peter Bevan. 2023. LAION POP: 600 000 High-Resolution Images With Detailed Descriptions. https:\/\/huggingface.co\/datasets\/laion\/laion-pop."},{"key":"e_1_2_1_45_1","volume-title":"Deep unsupervised learning using nonequilibrium thermodynamics. CoRR 1503","author":"Sohl-Dickstein Jascha","year":"2015","unstructured":"Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. CoRR 1503 (2015)."},{"key":"e_1_2_1_46_1","unstructured":"Jiaming Song Chenlin Meng and Stefano Ermon. 2021. Denoising diffusion implicit models. In ICLR. OpenReview.net."},{"key":"e_1_2_1_47_1","first-page":"13456","article-title":"Score-based generative modeling through stochastic differential equations","volume":"2011","author":"Song Yang","year":"2020","unstructured":"Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through stochastic differential equations. CoRR 2011 (2020), 13456.","journal-title":"CoRR"},{"key":"e_1_2_1_48_1","unstructured":"Stability. 2022a. Stable Diffusion v1.5 Model Card https:\/\/huggingface.co\/runwayml\/stable-diffusion-v1-5."},{"key":"e_1_2_1_49_1","unstructured":"Stability. 2022b. Stable Diffusion v2 Model Card Stable-Diffusion-2-Depth https:\/\/huggingface.co\/stabilityai\/stable-diffusion-2-depth."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2858238"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766960"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275054"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2988229"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3612404"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00317"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.299"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00191"},{"key":"e_1_2_1_58_1","volume-title":"ACM Transactions on Graphics (SIGGRAPH Asia 2018 issue) 37, 6 (Nov","author":"Xia Menghan","year":"2018","unstructured":"Menghan Xia, Xueting Liu, and Tien-Tsin Wong. 2018. Invertible Grayscale. ACM Transactions on Graphics (SIGGRAPH Asia 2018 issue) 37, 6 (Nov. 2018), 246:1--246:10."},{"key":"e_1_2_1_59_1","volume-title":"Invertible Image Rescaling","author":"Xiao Mingqing","unstructured":"Mingqing Xiao, Shuxin Zheng, Chang Liu, Yaolong Wang, Di He, Guolin Ke, Jiang Bian, Zhouchen Lin, and Tie-Yan Liu. 2020. Invertible Image Rescaling. Springer International Publishing, 126--144."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1703.03872"},{"key":"e_1_2_1_61_1","volume-title":"Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332","author":"Xu Xingqian","year":"2022","unstructured":"Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, and Humphrey Shi. 2022. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint arXiv:2211.08332 (2022)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2023.102091"},{"key":"e_1_2_1_63_1","unstructured":"Hu Ye Jun Zhang Sibo Liu Xiao Han and Wei Yang. 2023. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. (2023)."},{"key":"e_1_2_1_64_1","volume-title":"Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543","author":"Zhang Lvmin","year":"2023","unstructured":"Lvmin Zhang and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 (2023)."},{"key":"e_1_2_1_65_1","unstructured":"Xinyang Zhang Wentian Zhao Xin Lu and Jeff Chien. 2023. Text2Layer: Layered Image Generation using Latent Diffusion Model. arXiv:2307.09781 [cs.CV]"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.449"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658150","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3658150","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:54Z","timestamp":1750291554000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658150"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,19]]},"references-count":67,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7,19]]}},"alternative-id":["10.1145\/3658150"],"URL":"https:\/\/doi.org\/10.1145\/3658150","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,19]]},"assertion":[{"value":"2024-07-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}