{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T00:35:34Z","timestamp":1779928534088,"version":"3.53.1"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T00:00:00Z","timestamp":1690329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"GRF grant from the Research Grants Council of Hong Kong","award":["11205620"],"award-info":[{"award-number":["11205620"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>\n            Adjusting the photo color to associate with some design elements is an essential way for a graphic design to effectively deliver its message and make it aesthetically pleasing. However, existing tools and previous works face a dilemma between the ease of use and level of expressiveness. To this end, we introduce an interactive language-based approach for photo recoloring, which provides an intuitive system that can assist both experts and novices on graphic design. Given a graphic design containing a photo that needs to be recolored, our model can predict the source colors and the target regions, and then recolor the target regions with the source colors based on the given language-based instruction. The multi-granularity of the instruction allows diverse user intentions. The proposed novel task faces several unique challenges, including:\n            <jats:italic>1) color accuracy<\/jats:italic>\n            for recoloring with exactly the same color from the target design element as specified by the user;\n            <jats:italic>2) multi-granularity instructions<\/jats:italic>\n            for parsing instructions correctly to generate a specific result or multiple plausible ones; and\n            <jats:italic>3) locality<\/jats:italic>\n            for recoloring in semantically meaningful local regions to preserve original image semantics. To address these challenges, we propose a model called\n            <jats:italic>LangRecol<\/jats:italic>\n            with two main components: the language-based source color prediction module and the semantic-palette-based photo recoloring module. We also introduce an approach for generating a synthetic graphic design dataset with instructions to enable model training. We evaluate our model via extensive experiments and user studies. We also discuss several practical applications, showing the effectiveness and practicality of our approach. Please find the code and data at https:\/\/zhenwwang.github.io\/langrecol.\n          <\/jats:p>","DOI":"10.1145\/3592111","type":"journal-article","created":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T15:47:45Z","timestamp":1690386465000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":36,"title":["Language-based Photo Color Adjustment for Graphic Designs"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0215-660X","authenticated-orcid":false,"given":"Zhenwei","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Computer Science, City University of Hong Kong, Hong Kong, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4007-2776","authenticated-orcid":false,"given":"Nanxuan","family":"Zhao","sequence":"additional","affiliation":[{"name":"Adobe Research, San Jose, California, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2388-3542","authenticated-orcid":false,"given":"Gerhard","family":"Hancke","sequence":"additional","affiliation":[{"name":"Department of Computer Science, City University of Hong Kong, Hong Kong, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8957-8129","authenticated-orcid":false,"given":"Rynson W. H.","family":"Lau","sequence":"additional","affiliation":[{"name":"Department of Computer Science, City University of Hong Kong, Hong Kong, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,7,26]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"CAMS: Color-Aware Multi-Style Transfer. arXiv preprint arXiv:2106.13920","author":"Afifi Mahmoud","year":"2021","unstructured":"Mahmoud Afifi, Abdullah Abuolaim, Mostafa Hussien, Marcus A Brubaker, and Michael S Brown. 2021a. CAMS: Color-Aware Multi-Style Transfer. arXiv preprint arXiv:2106.13920 (2021)."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00785"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3092919.3092926"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3002176"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201275"},{"key":"e_1_2_2_6_1","volume-title":"Retrieved","author":"Aland Maggie","year":"2017","unstructured":"Maggie Aland. 2017. 10 tips and ideas to make a flyer that stands out. Retrieved Jan 03, 2022 from https:\/\/www.lucidpress.com\/blog\/10-creative-ways-to-make-flyer-stand-out"},{"key":"e_1_2_2_7_1","volume-title":"Retrieved","author":"ANL","year":"2022","unstructured":"ANL [n. d.]. Guide to Effective Poster Design. Retrieved Jan 03, 2022 from https:\/\/www.anl.gov\/education\/guide-to-effective-poster-design"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2016.12.005"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141911.1141935"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01258-8_27"},{"key":"e_1_2_2_11_1","volume-title":"Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727","author":"Bau David","year":"2020","unstructured":"David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. 2020. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727 (2020)."},{"key":"e_1_2_2_12_1","volume-title":"Basic color terms: Their universality and evolution","author":"Berlin Brent","unstructured":"Brent Berlin and Paul Kay. 1991. Basic color terms: Their universality and evolution. Univ of California Press."},{"key":"e_1_2_2_13_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901."},{"key":"e_1_2_2_14_1","volume-title":"Retrieved","author":"Canva","year":"2022","unstructured":"Canva [n. d.]. The best Google Font combinations to try. Retrieved May 06, 2022 from https:\/\/www.canva.com\/learn\/best-google-font-combinations\/"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766978"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1077399.1077408"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00909"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2682628"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394171.3413551"},{"key":"e_1_2_2_20_1","volume-title":"Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.","author":"Cho Kyunghyun","year":"2014","unstructured":"Kyunghyun Cho, Bart Van Merri\u00ebnboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1179352.1141933"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.608"},{"key":"e_1_2_2_23_1","volume-title":"Deepprop: Extracting deep features from a single image for edit propagation. In Computer Graphics Forum","author":"Endo Yuki","year":"2016","unstructured":"Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani. 2016. Deepprop: Extracting deep features from a single image for edit propagation. In Computer Graphics Forum, Vol. 35. Wiley Online Library, 189--201."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01525"},{"key":"e_1_2_2_25_1","volume-title":"Generative adversarial nets. Advances in neural information processing systems 27","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014)."},{"key":"e_1_2_2_26_1","volume-title":"Retrieved","year":"2020","unstructured":"GraphicsZoo 2020. 15 Tips to Create Unique Poster Design for Your Brand. Retrieved Feb 16, 2022 from https:\/\/www.graphicszoo.com\/article\/15-tips-to-create-unique-poster-design-for-your-brand"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461997"},{"key":"e_1_2_2_28_1","first-page":"1","article-title":"Deep exemplar-based colorization","volume":"37","author":"He Mingming","year":"2018","unstructured":"Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--16.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.2502935"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101149.1101223"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01354"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1108\/02651330910950411"},{"key":"e_1_2_2_33_1","volume-title":"Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276","author":"Kawar Bahjat","year":"2022","unstructured":"Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2022. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 (2022)."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00153"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01440"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/col.22144"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481301"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1186562.1015780"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00790"},{"key":"e_1_2_2_40_1","first-page":"22020","article-title":"Lightweight generative adversarial networks for text-guided image manipulation","volume":"33","author":"Li Bowen","year":"2020","unstructured":"Bowen Li, Xiaojuan Qi, Philip Torr, and Thomas Lukasiewicz. 2020b. Lightweight generative adversarial networks for text-guided image manipulation. Advances in Neural Information Processing Systems 33 (2020), 22020--22031.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_41_1","volume-title":"Computer Graphics Forum","author":"Li Yuanzhen","unstructured":"Yuanzhen Li, Edward Adelson, and Aseem Agarwala. 2008. ScribbleBoost: Adding Classification to Edge-Aware Interpolation of Local Image and Video Adjustments. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 1255--1264."},{"key":"e_1_2_2_42_1","volume-title":"Proceedings, Part XI 16","author":"Liu Xihui","year":"2020","unstructured":"Xihui Liu, Zhe Lin, Jianming Zhang, Handong Zhao, Quan Tran, Xiaogang Wang, and Hongsheng Li. 2020. Open-edit: Open-domain image manipulation with open-vocabulary instructions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16. Springer, 89--106."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.740"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/2383847.2383887"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00695"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275035"},{"key":"e_1_2_2_47_1","volume-title":"Retrieved","author":"Mcguire Sara","year":"2019","unstructured":"Sara Mcguire. 2019. Poster Design Guide: How to Make an Eye-Catching Poster in 2020. Retrieved Jan 03, 2022 from https:\/\/venngage.com\/blog\/poster-design\/"},{"key":"e_1_2_2_48_1","volume-title":"Proceedings of the 32nd International Conference on Neural Information Processing Systems. 42--51","author":"Nam Seonghyeon","year":"2018","unstructured":"Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. 2018. Text-adaptive generative adversarial networks: manipulating images with natural language. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 42--51."},{"key":"e_1_2_2_49_1","volume-title":"Computer Graphics Forum","author":"Nguyen Rang MH","unstructured":"Rang MH Nguyen, Brian Price, Scott Cohen, and Michael S Brown. 2017. Group-Theme Recoloring for Multi-Image Color Consistency. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 83--92."},{"key":"e_1_2_2_50_1","volume-title":"Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741","author":"Nichol Alex","year":"2021","unstructured":"Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702149"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.48"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00209"},{"key":"e_1_2_2_54_1","volume-title":"International Conference on Machine Learning. PMLR, 8748--8763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748--8763."},{"key":"e_1_2_2_55_1","volume-title":"Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125","author":"Ramesh Aditya","year":"2022","unstructured":"Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/38.946629"},{"key":"e_1_2_2_57_1","volume-title":"Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.","author":"Saharia Chitwan","year":"2022","unstructured":"Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv preprint arXiv:2205.11487 (2022)."},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275054"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2988229"},{"key":"e_1_2_2_60_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_2_2_61_1","volume-title":"Computer Graphics Forum","author":"Wang Yili","unstructured":"Yili Wang, Yifan Liu, and Kun Xu. 2019. An Improved Geometric Approach for Palette-based Image Decomposition and Recoloring. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 11--22."},{"key":"e_1_2_2_62_1","volume-title":"Proceedings, Part XV. Springer, 271--288","author":"Wang Yi","year":"2022","unstructured":"Yi Wang, Menghan Xia, Lu Qi, Jing Shao, and Yu Qiao. 2022. PalGAN: Image Colorization with Palette Generative Adversarial Networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XV. Springer, 271--288."},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i3.20170"},{"key":"e_1_2_2_64_1","volume-title":"Retrieved","author":"West Chloe","year":"2020","unstructured":"Chloe West. 2020. The Ultimate Guide to Flyer Design. Retrieved May 02, 2022 from https:\/\/visme.co\/blog\/flyer-design\/"},{"key":"e_1_2_2_65_1","volume-title":"ISCC Annual Meeting. Citeseer.","author":"Geoff","unstructured":"Geoff Woolfe et al. 2007. Natural language color editing. In ISCC Annual Meeting. Citeseer."},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01023"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00229"},{"key":"e_1_2_2_68_1","volume-title":"Sketch-HairSalon: Deep Sketch-based Hair Image Synthesis. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2021) 40, 6","author":"Xiao Chufeng","year":"2021","unstructured":"Chufeng Xiao, Deng Yu, Xiaoguang Han, Youyi Zheng, and Hongbo Fu. 2021. Sketch-HairSalon: Deep Sketch-based Hair Image Synthesis. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2021) 40, 6 (2021), 1--16."},{"key":"e_1_2_2_69_1","first-page":"1","article-title":"Efficient affinity-based edit propagation using kd tree","volume":"28","author":"Xu Kun","year":"2009","unstructured":"Kun Xu, Yong Li, Tao Ju, Shi-Min Hu, and Tian-Qiang Liu. 2009. Efficient affinity-based edit propagation using kd tree. ACM Transactions on Graphics (TOG) 28, 5 (2009), 1--6.","journal-title":"ACM Transactions on Graphics (TOG)"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2005.864231"},{"key":"e_1_2_2_71_1","volume-title":"Wide residual networks. arXiv preprint arXiv:1605.07146","author":"Zagoruyko Sergey","year":"2016","unstructured":"Sergey Zagoruyko and Nikos Komodakis. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)."},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2671779"},{"key":"e_1_2_2_73_1","volume-title":"Real-time user-guided image colorization with learned deep priors. arXiv preprint arXiv:1705.02999","author":"Zhang Richard","year":"2017","unstructured":"Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017b. Real-time user-guided image colorization with learned deep priors. arXiv preprint arXiv:1705.02999 (2017)."},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447647"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.186"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356561"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592111","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3592111","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:45Z","timestamp":1750178265000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592111"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,26]]},"references-count":76,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.1145\/3592111"],"URL":"https:\/\/doi.org\/10.1145\/3592111","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,26]]},"assertion":[{"value":"2023-07-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}