{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T16:05:37Z","timestamp":1778256337988,"version":"3.51.4"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T00:00:00Z","timestamp":1669766400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Hong Kong Research Grants Council (RGC) GRF Scheme","award":["CityU 11216122"],"award-info":[{"award-number":["CityU 11216122"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:p>\n            We propose the first unified framework\n            <jats:italic>UniColor<\/jats:italic>\n            to support colorization in multiple modalities, including both unconditional and conditional ones, such as stroke, exemplar, text, and even a mix of them. Rather than learning a separate model for each type of condition, we introduce a two-stage colorization framework for incorporating various conditions into a single model. In the first stage, multi-modal conditions are converted into a common representation of hint points. Particularly, we propose a novel CLIP-based method to convert the text to hint points. In the second stage, we propose a Transformer-based network composed of\n            <jats:italic>Chroma-VQGAN<\/jats:italic>\n            and\n            <jats:italic>Hybrid-Transformer<\/jats:italic>\n            to generate diverse and high-quality colorization results conditioned on hint points. Both qualitative and quantitative comparisons demonstrate that our method outperforms state-of-the-art methods in every control modality and further enables multi-modal colorization that was not feasible before. Moreover, we design an interactive interface showing the effectiveness of our unified framework in practical usage, including automatic colorization, hybrid-control colorization, local recolorization, and iterative color editing. Our code and models are available at\n            <jats:italic>https:\/\/luckyhzt.github.io\/unicolor<\/jats:italic>\n            .\n          <\/jats:p>","DOI":"10.1145\/3550454.3555471","type":"journal-article","created":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T21:19:07Z","timestamp":1669843147000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":42,"title":["UniColor"],"prefix":"10.1145","volume":"41","author":[{"given":"Zhitong","family":"Huang","sequence":"first","affiliation":[{"name":"City University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nanxuan","family":"Zhao","sequence":"additional","affiliation":[{"name":"University of Bath, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Liao","sequence":"additional","affiliation":[{"name":"City University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,11,30]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.120"},{"key":"e_1_2_2_2_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations, ICLR","author":"Bahdanau Dzmitry","year":"2015","unstructured":"Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015."},{"key":"e_1_2_2_3_1","first-page":"443","article-title":"Coloring with Words","volume":"2018","author":"Bahng Hyojin","year":"2018","unstructured":"Hyojin Bahng, Seungjoo Yoo, Wonwoong Cho, David Keetae Park, Ziming Wu, Xiaojuan Ma, and Jaegul Choo. 2018. Coloring with Words: Guiding Image Colorization Through Text-Based Palette Generation. In ECCV 2018. 443--459.","journal-title":"Guiding Image Colorization Through Text-Based Palette Generation. In ECCV"},{"key":"e_1_2_2_4_1","doi-asserted-by":"crossref","unstructured":"Yun Cao Zhiming Zhou Weinan Zhang and Yong Yu. 2017. Unsupervised Diverse Colorization via Generative Adversarial Networks. In Machine Learning and Knowledge Discovery in Databases. 151--166.","DOI":"10.1007\/978-3-319-71249-9_10"},{"key":"e_1_2_2_5_1","volume-title":"Language-Based Image Editing with Recurrent Attentive Models. 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Chen Jianbo","year":"2018","unstructured":"Jianbo Chen, Yelong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. 2018. Language-Based Image Editing with Recurrent Attentive Models. 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (2018), 8721--8729."},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"1703","author":"Chen Mark","year":"2020","unstructured":"Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative Pretraining From Pixels. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, 1691--1703."},{"key":"e_1_2_2_7_1","volume-title":"Generating Long Sequences with Sparse Transformers. URL https:\/\/openai.com\/blog\/sparse-transformers","author":"Child Rewon","year":"2019","unstructured":"Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating Long Sequences with Sparse Transformers. URL https:\/\/openai.com\/blog\/sparse-transformers (2019)."},{"key":"e_1_2_2_8_1","volume-title":"Deoldify: A Deep Learning based Project for Colorizing and Restoring Old Images and Video. https:\/\/github.com\/jantic\/DeOldify.","year":"2018","unstructured":"Deoldify. 2018. Deoldify: A Deep Learning based Project for Colorizing and Restoring Old Images and Video. https:\/\/github.com\/jantic\/DeOldify."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.307"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n19-1423"},{"key":"e_1_2_2_11_1","volume-title":"9th International Conference on Learning Representations, ICLR 2021","author":"Dosovitskiy Alexey","year":"2021","unstructured":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3--7, 2021."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12822"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01268"},{"key":"e_1_2_2_14_1","volume-title":"Advances in Neural Information Processing Systems","volume":"27","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Vol. 27."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.477378"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201365"},{"key":"e_1_2_2_17_1","volume-title":"Advances in Neural Information Processing Systems","volume":"30","author":"Heusel Martin","year":"2017","unstructured":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, Vol. 30."},{"key":"e_1_2_2_18_1","first-page":"6840","article-title":"Denoising Diffusion Probabilistic Models","volume":"33","author":"Ho Jonathan","year":"2020","unstructured":"Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020a. Denoising Diffusion Probabilistic Models. In Advances in Neural Information ProcessingSystems, Vol. 33. 6840--6851.","journal-title":"Advances in Neural Information ProcessingSystems"},{"key":"e_1_2_2_19_1","unstructured":"Jonathan Ho Nal Kalchbrenner Dirk Weissenborn and Tim Salimans. 2020b. Axial Attention in Multidimensional Transformers. https:\/\/openreview.net\/forum?id=H1e5GJBtDr"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925974"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.632"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3481544"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1086"},{"key":"e_1_2_2_24_1","volume-title":"Image Colorization Using Generative Adversarial Networks and Transfer Learning. 2020 International Conference on Machine Vision and Image Processing (MVIP)","author":"Kiani Leila","year":"2020","unstructured":"Leila Kiani, Masoudnia Saeed, and Hossein Nezamabadi-pour. 2020. Image Colorization Using Generative Adversarial Networks and Transfer Learning. 2020 International Conference on Machine Vision and Image Processing (MVIP) (2020), 1--6."},{"key":"e_1_2_2_25_1","volume-title":"Rush","author":"Kim Yoon","year":"2017","unstructured":"Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. 2017. Structured Attention Networks. CoRR abs\/1702.00887 (2017). arXiv:1702.00887"},{"key":"e_1_2_2_26_1","volume-title":"Colorization Transformer. In International Conference on Learning Representations.","author":"Kumar Manoj","year":"2021","unstructured":"Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. 2021. Colorization Transformer. In International Conference on Learning Representations."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1186562.1015780"},{"key":"e_1_2_2_28_1","first-page":"740","article-title":"Microsoft COCO","volume":"2014","author":"Lin Tsung-Yi","year":"2014","unstructured":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision - ECCV 2014. 740--755.","journal-title":"Common Objects in Context. In Computer Vision - ECCV"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","unstructured":"Qing Luan Fang Wen Daniel Cohen-Or Lin Liang Ying-Qing Xu and Heung-Yeung Shum. 2007. Natural Image Colorization. In Rendering Techniques. 10.2312\/EGWR\/EGSR07\/309-320","DOI":"10.2312\/EGWR\/EGSR07\/309-320"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01005"},{"key":"e_1_2_2_31_1","doi-asserted-by":"crossref","unstructured":"Varun Manjunatha Mohit Iyyer Jordan Boyd-Graber and Larry Davis. 2018. Learning to Color from Language. In North American Chapter of the Association for Computational Linguistics.","DOI":"10.18653\/v1\/N18-2120"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01264-9_47"},{"key":"e_1_2_2_33_1","volume-title":"Schwing","author":"Messaoud Safa","year":"2018","unstructured":"Safa Messaoud, David A. Forsyth, and Alexander G. Schwing. 2018. Structural Consistency and Controllability for Diverse Colorization. In ECCV (6). 603--619."},{"key":"e_1_2_2_34_1","volume-title":"Image Transformer. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"4064","author":"Parmar Niki","year":"2018","unstructured":"Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image Transformer. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). PMLR, 4055--4064."},{"key":"e_1_2_2_35_1","volume-title":"Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML."},{"key":"e_1_2_2_36_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"8831","author":"Ramesh Aditya","year":"2021","unstructured":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139). 8821--8831."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2111.05826"},{"key":"e_1_2_2_39_1","volume-title":"3rd International Conference on Learning Representations, ICLR","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings."},{"key":"e_1_2_2_40_1","volume-title":"Instance-aware Image Colorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Su Jheng-Wei","year":"2020","unstructured":"Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang. 2020. Instance-aware Image Colorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_41_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, Vol. 30."},{"key":"e_1_2_2_42_1","volume-title":"ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution. In The IEEE Winter Conference on Applications of Computer Vision. 2445--2454","author":"Vitoria Patricia","year":"2020","unstructured":"Patricia Vitoria, Lara Raad, and Coloma Ballester. 2020. ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution. In The IEEE Winter Conference on Applications of Computer Vision. 2445--2454."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01139"},{"key":"e_1_2_2_45_1","volume-title":"Scaling Autoregressive Video Models. In International Conference on Learning Representations.","author":"Weissenborn Dirk","year":"2020","unstructured":"Dirk Weissenborn, Oscar T\u00e4ckstr\u00f6m, and Jakob Uszkoreit. 2020. Scaling Autoregressive Video Models. In International Conference on Learning Representations."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","unstructured":"Chenfei Wu Jian Liang Lei Ji Fan Yang Yuejian Fang Daxin Jiang and Nan Duan. 2021a. N\u00dcWA: Visual Synthesis Pre-training for Neural visUal World creAtion. 10.48550\/ARXIV.2111.12417","DOI":"10.48550\/ARXIV.2111.12417"},{"key":"e_1_2_2_47_1","volume-title":"Towards Vivid and Diverse Image Colorization with Generative Color Prior. In International Conference on Computer Vision (ICCV).","author":"Wu Yanze","year":"2021","unstructured":"Yanze Wu, Xintao Wang, Yu Li, Honglun Zhang, Xun Zhao, and Ying Shan. 2021b. Towards Vivid and Diverse Image Colorization with Generative Color Prior. In International Conference on Computer Vision (ICCV)."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8683686"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00938"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00824"},{"key":"e_1_2_2_51_1","doi-asserted-by":"crossref","unstructured":"Richard Zhang Phillip Isola and Alexei A Efros. 2016. Colorful Image Colorization. In ECCV.","DOI":"10.1007\/978-3-319-46487-9_40"},{"key":"e_1_2_2_52_1","doi-asserted-by":"crossref","unstructured":"Richard Zhang Phillip Isola Alexei A Efros Eli Shechtman and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_53_1","article-title":"Real-Time User-Guided Image Colorization with Learned Deep Priors","volume":"9","author":"Zhang Richard","year":"2017","unstructured":"Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image Colorization with Learned Deep Priors. ACM Transactions on Graphics (TOG) 9, 4 (2017).","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_2_2_54_1","unstructured":"Zhu Zhang Jianxin Ma Chang Zhou Rui Men Zhikang Li Ming Ding Jie Tang Jingren Zhou and Hongxia Yang. 2021. UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis. In Advances in Neural Information Processing Systems."},{"key":"e_1_2_2_55_1","volume-title":"Lau","author":"Zhao Nanxuan","year":"2021","unstructured":"Nanxuan Zhao, Quanlong Zheng, Jing Liao, Ying Cao, Hanspeter Pfister, and Rynson W.H. Lau. 2021. Selective Region-based Photo Color Adjustment for Graphic Designs. ACM Transactions on Graphics 40 (2021). Issue 2."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550454.3555471","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3550454.3555471","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:51:43Z","timestamp":1750182703000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550454.3555471"}},"subtitle":["A Unified Framework for Multi-Modal Colorization with Transformer"],"short-title":[],"issued":{"date-parts":[[2022,11,30]]},"references-count":55,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["10.1145\/3550454.3555471"],"URL":"https:\/\/doi.org\/10.1145\/3550454.3555471","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,30]]},"assertion":[{"value":"2022-11-30","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}