{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T13:42:55Z","timestamp":1762868575818,"version":"3.40.5"},"reference-count":48,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T00:00:00Z","timestamp":1626307200000},"content-version":"vor","delay-in-days":14,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Computer Graphics Forum"],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Despite recent breakthroughs in deep learning methods for image lighting enhancement, they are inferior when applied to portraits because 3D facial information is ignored in their models. To address this, we present a novel deep learning framework for portrait lighting enhancement based on 3D facial guidance. Our framework consists of two stages. In the first stage, corrected lighting parameters are predicted by a network from the input bad lighting image, with the assistance of a 3D morphable model and a differentiable renderer. Given the predicted lighting parameter, the differentiable renderer renders a face image with corrected shading and texture, which serves as the 3D guidance for learning image lighting enhancement in the second stage. To better exploit the long\u2010range correlations between the input and the guidance, in the second stage, we design an image\u2010to\u2010image translation network with a novel transformer architecture, which automatically produces a lighting\u2010enhanced result. Experimental results on the FFHQ dataset and in\u2010the\u2010wild images show that the proposed method outperforms state\u2010of\u2010the\u2010art methods in terms of both quantitative metrics and visual quality.<\/jats:p>","DOI":"10.1111\/cgf.14350","type":"journal-article","created":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T10:45:45Z","timestamp":1626345945000},"page":"177-188","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Deep Portrait Lighting Enhancement with 3D Guidance"],"prefix":"10.1111","volume":"40","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3641-1667","authenticated-orcid":false,"given":"Fangzhou","family":"Han","sequence":"first","affiliation":[{"name":"City University of Hong Kong"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5102-1464","authenticated-orcid":false,"given":"Can","family":"Wang","sequence":"additional","affiliation":[{"name":"City University of Hong Kong"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1330-4532","authenticated-orcid":false,"given":"Hao","family":"Du","sequence":"additional","affiliation":[{"name":"City University of Hong Kong"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7014-5377","authenticated-orcid":false,"given":"Jing","family":"Liao","sequence":"additional","affiliation":[{"name":"City University of Hong Kong"}]}],"member":"311","published-online":{"date-parts":[[2021,7,15]]},"reference":[{"key":"e_1_2_7_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2794218"},{"key":"e_1_2_7_3_2","unstructured":"ChenW. LingH. GaoJ. SmithE. LehtinenJ. JacobsonA. FidlerS.: Learning to predict 3d objects with an interpolation-based differentiable renderer. InAdvances in Neural Information Processing Systems(2019) pp.9609\u20139619. 3"},{"key":"e_1_2_7_4_2","doi-asserted-by":"crossref","unstructured":"CarionN. MassaF. SynnaeveG. UsunierN. KirillovA. ZagoruykoS.: End-to-end object detection with transformers. InEuropean Conference on Computer Vision(2020) Springer pp.213\u2013229. 2 3","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"e_1_2_7_5_2","doi-asserted-by":"crossref","unstructured":"ChenY.-S. WangY.-C. KaoM.-H. ChuangY.-Y.: Deep photo enhancer: Unpaired learning for image enhancement from photographs with gans. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) pp.6306\u20136314. 2","DOI":"10.1109\/CVPR.2018.00660"},{"key":"e_1_2_7_6_2","unstructured":"DosovitskiyA. BeyerL. KolesnikovA. WeissenbornD. ZhaiX. UnterthinerT. DehghaniM. MindererM. HeigoldG. GellyS. et al.: An image is worth 16\u00d716 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929(2020). 2 3"},{"key":"e_1_2_7_7_2","doi-asserted-by":"crossref","unstructured":"DengY. LoyC. C. TangX.: Aesthetic-driven image enhancement by adversarial learning. InProceedings of the 26th ACM international conference on Multimedia(2018) pp.870\u2013878. 2","DOI":"10.1145\/3240508.3240531"},{"key":"e_1_2_7_8_2","doi-asserted-by":"crossref","unstructured":"DengY. YangJ. XuS. ChenD. JiaY. TongX.: Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. InProceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops(2019) pp.0\u20130. 3","DOI":"10.1109\/CVPRW.2019.00038"},{"key":"e_1_2_7_9_2","doi-asserted-by":"crossref","unstructured":"FourureD. EmonetR. FromontE. MuseletD. TremeauA. WolfC.: Residual conv-deconv grid network for semantic segmentation.arXiv preprint arXiv:1707.07958(2017). 3","DOI":"10.5244\/C.31.181"},{"key":"e_1_2_7_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925933"},{"key":"e_1_2_7_11_2","doi-asserted-by":"crossref","unstructured":"GenovaK. ColeF. MaschinotA. SarnaA. VlasicD. FreemanW. T.: Unsupervised training for 3d morphable model regression. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) pp.8377\u20138386. 3","DOI":"10.1109\/CVPR.2018.00874"},{"key":"e_1_2_7_12_2","doi-asserted-by":"crossref","unstructured":"GuoC. LiC. GuoJ. LoyC. C. HouJ. KwongS. CongR.: Zero-reference deep curve estimation for low-light image enhancement. InProceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition(2020) pp.1780\u20131789. 2 8","DOI":"10.1109\/CVPR42600.2020.00185"},{"key":"e_1_2_7_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2009.08.002"},{"key":"e_1_2_7_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3181974"},{"key":"e_1_2_7_15_2","doi-asserted-by":"crossref","unstructured":"HeK. ZhangX. RenS. SunJ.: Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition(2016) pp.770\u2013778. 6","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_7_16_2","doi-asserted-by":"crossref","unstructured":"HouA. ZhangZ. SarkisM. BiN. TongY. LiuX.: Towards high fidelity face relighting with realistic shadows.arXiv preprint arXiv:2104.00825(2021). 8","DOI":"10.1109\/CVPR46437.2021.01448"},{"key":"e_1_2_7_17_2","doi-asserted-by":"crossref","unstructured":"IsolaP. ZhuJ.-Y. ZhouT. EfrosA. A.: Image-to-image translation with conditional adversarial networks. InProceedings of the IEEE conference on computer vision and pattern recognition(2017) pp.1125\u20131134. 5 6","DOI":"10.1109\/CVPR.2017.632"},{"key":"e_1_2_7_18_2","unstructured":"JiangY. GongX. LiuD. ChengY. FangC. ShenX. YangJ. ZhouP. WangZ.: Enlightengan: Deep light enhancement without paired supervision.arXiv preprint arXiv:1906.06972(2019). 2 8"},{"key":"e_1_2_7_19_2","doi-asserted-by":"crossref","unstructured":"KarrasT. LaineS. AilaT.: A style-based generator architecture for generative adversarial networks. InProceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition(2019) pp.4401\u20134410. 2 5","DOI":"10.1109\/CVPR.2019.00453"},{"key":"e_1_2_7_20_2","doi-asserted-by":"crossref","unstructured":"Konstantin Sofiiuk Polina Popenova A. K.: Foreground-aware semantic representations for image harmonization.arXiv preprint arXiv:2006.00809(2020). 9","DOI":"10.1109\/WACV48630.2021.00166"},{"key":"e_1_2_7_21_2","doi-asserted-by":"crossref","unstructured":"KatoH. UshikuY. HaradaT.: Neural 3d mesh renderer. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) pp.3907\u20133916. 3","DOI":"10.1109\/CVPR.2018.00411"},{"key":"e_1_2_7_22_2","doi-asserted-by":"crossref","unstructured":"LoperM. M. BlackM. J.: Opendr: An approximate differentiable renderer. InEuropean Conference on Computer Vision(2014) Springer pp.154\u2013169. 3","DOI":"10.1007\/978-3-319-10584-0_11"},{"key":"e_1_2_7_23_2","doi-asserted-by":"crossref","unstructured":"LiuS. LiT. ChenW. LiH.: Soft rasterizer: A differentiable renderer for image-based 3d reasoning. InProceedings of the IEEE International Conference on Computer Vision(2019) pp.7708\u20137717. 2 3 4","DOI":"10.1109\/ICCV.2019.00780"},{"key":"e_1_2_7_24_2","doi-asserted-by":"crossref","unstructured":"MurmannL. GharbiM. AittalaM. DurandF.: A dataset of multi-illumination images in the wild. InProceedings of the IEEE\/CVF International Conference on Computer Vision(2019) pp.4080\u20134089. 2","DOI":"10.1109\/ICCV.2019.00418"},{"key":"e_1_2_7_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323027"},{"key":"e_1_2_7_26_2","unstructured":"MallyaA. WangT.-C. SapraK. LiuM.-Y.: World-consistent video-to-video synthesis.arXiv preprint arXiv:2007.08509(2020). 5"},{"key":"e_1_2_7_27_2","unstructured":"NietoG. JiddiS. RobertP.: Robust point light source estimation using differentiable rendering.arXiv preprint arXiv:1812.04857(2018). 3"},{"key":"e_1_2_7_28_2","doi-asserted-by":"crossref","unstructured":"NestmeyerT. LalondeJ.-F. MatthewsI. LehrmannA.: Learning physics-guided face relighting under directional light. InProceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition(2020) pp.5124\u20135133. 3","DOI":"10.1109\/CVPR42600.2020.00517"},{"key":"e_1_2_7_29_2","doi-asserted-by":"crossref","unstructured":"PaysanP. KnotheR. AmbergB. RomdhaniS. VetterT.: A 3d face model for pose and illumination invariant face recognition. In2009 sixth IEEE international conference on advanced video and signal based surveillance(2009) Ieee pp.296\u2013301. 4","DOI":"10.1109\/AVSS.2009.58"},{"key":"e_1_2_7_30_2","doi-asserted-by":"crossref","unstructured":"ParkT. LiuM.-Y. WangT.-C. ZhuJ.-Y.: Semantic image synthesis with spatially-adaptive normalization. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2019) pp.2337\u20132346. 5","DOI":"10.1109\/CVPR.2019.00244"},{"key":"e_1_2_7_31_2","doi-asserted-by":"crossref","unstructured":"Pavla kosG. ZhuL. ZhouX. DaniilidisK.: Learning to estimate 3d human pose and shape from a single color image. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) pp.459\u2013468. 3","DOI":"10.1109\/CVPR.2018.00055"},{"key":"e_1_2_7_32_2","doi-asserted-by":"crossref","unstructured":"RonnebergerO. FischerP. BroxT.: U-net: Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing and computer-assisted intervention(2015) Springer pp.234\u2013241. 2 3","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_2_7_33_2","doi-asserted-by":"crossref","unstructured":"RhodinH. RobertiniN. RichardtC. SeidelH.-P. TheobaltC.: A versatile scene model with differentiable visibility applied to generative pose estimation. InProceedings of the IEEE International Conference on Computer Vision(2015) pp.765\u2013773. 3","DOI":"10.1109\/ICCV.2015.94"},{"key":"e_1_2_7_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323008"},{"key":"e_1_2_7_35_2","doi-asserted-by":"crossref","unstructured":"SenguptaS. KanazawaA. CastilloC. D. JacobsD. W.: Sfsnet: Learning shape reflectance and illuminance of facesin the wild'. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition(2018) pp.6296\u20136305. 3","DOI":"10.1109\/CVPR.2018.00659"},{"key":"e_1_2_7_36_2","doi-asserted-by":"crossref","unstructured":"ShuZ. YumerE. HadapS. SunkavalliK. ShechtmanE. SamarasD.: Neural face editing with intrinsic image disentangling. InProceedings of the IEEE conference on computer vision and pattern recognition(2017) pp.5541\u20135550. 3","DOI":"10.1109\/CVPR.2017.578"},{"key":"e_1_2_7_37_2","doi-asserted-by":"crossref","unstructured":"TulsianiS. ZhouT. EfrosA. A. MalikJ.: Multi-view supervision for single-view reconstruction via differentiable ray consistency. InProceedings of the IEEE conference on computer vision and pattern recognition(2017) pp.2626\u20132634. 3","DOI":"10.1109\/CVPR.2017.30"},{"key":"e_1_2_7_38_2","unstructured":"VaswaniA. ShazeerN. ParmarN. UszkoreitJ. JonesL. GomezA. N. KaiserL. PolosukhinI.: Attention is all you need.arXiv preprint arXiv:1706.03762(2017). 2 3"},{"key":"e_1_2_7_39_2","doi-asserted-by":"crossref","unstructured":"WangT.-C. LiuM.-Y. ZhuJ.-Y. TaoA. KautzJ. CatanzaroB.: High-resolution image synthesis and semantic manipulation with conditional gans. InProceedings of the IEEE conference on computer vision and pattern recognition(2018) pp.8798\u20138807. 5","DOI":"10.1109\/CVPR.2018.00917"},{"key":"e_1_2_7_40_2","unstructured":"WeiC. WangW. YangW. LiuJ.: Deep retinex decomposition for low-light enhancement.arXiv preprint arXiv:1808.04560(2018). 2"},{"key":"e_1_2_7_41_2","doi-asserted-by":"crossref","unstructured":"WanZ. ZhangJ. ChenD. LiaoJ.: High-fidelity pluralistic image completion with transformers.arXiv preprint arXiv:2103.14031(2021). 3","DOI":"10.1109\/ICCV48922.2021.00465"},{"key":"e_1_2_7_42_2","doi-asserted-by":"crossref","unstructured":"WangR. ZhangQ. FuC.-W. ShenX. ZhengW.-S. JiaJ.: Underexposed photo enhancement using deep illumination estimation. InProceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition(2019) pp.6849\u20136857. 2","DOI":"10.1109\/CVPR.2019.00701"},{"key":"e_1_2_7_43_2","unstructured":"YuR. LiuW. ZhangY. QuZ. ZhaoD. ZhangB.: Deepexposure: Learning to expose photos with asynchronously reinforced adversarial learning. InAdvances in Neural Information Processing Systems(2018) pp.2149\u20132159. 2"},{"key":"e_1_2_7_44_2","doi-asserted-by":"crossref","unstructured":"YuanL. SunJ.: Automatic exposure correction of consumer photographs. InEuropean Conference on Computer Vision(2012) Springer pp.771\u2013785. 2","DOI":"10.1007\/978-3-642-33765-9_55"},{"key":"e_1_2_7_45_2","unstructured":"YanX. YangJ. YumerE. GuoY. LeeH.: Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. InAdvances in neural information processing systems(2016) pp.1696\u20131704. 3"},{"key":"e_1_2_7_46_2","unstructured":"ZhouH. HadapS. SunkavalliK. JacobsD. W.: Deep single-image portrait relighting. InProceedings of the IEEE\/CVF International Conference on Computer Vision(2019) pp.7194\u20137202. 2 3 6 8"},{"key":"e_1_2_7_47_2","doi-asserted-by":"crossref","unstructured":"ZhuJ.-Y. ParkT. IsolaP. EfrosA. A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. InProceedings of the IEEE international conference on computer vision(2017) pp.2223\u20132232. 2","DOI":"10.1109\/ICCV.2017.244"},{"key":"e_1_2_7_48_2","unstructured":"ZhangX. C. TsaiY.-T. PandeyR. ZhangX. NgR. JacobsD. E. et al.: Portrait shadow manipulation.arXiv preprint arXiv:2005.08925(2020). 3"},{"key":"e_1_2_7_49_2","unstructured":"ZhouY. XiaoQ.: Gan-based facial attractiveness enhancement.arXiv preprint arXiv:2006.02766(2020). 2"}],"container-title":["Computer Graphics Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/cgf.14350","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/cgf.14350","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/cgf.14350","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T05:04:50Z","timestamp":1693285490000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/cgf.14350"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.1111\/cgf.14350"],"URL":"https:\/\/doi.org\/10.1111\/cgf.14350","archive":["Portico"],"relation":{},"ISSN":["0167-7055","1467-8659"],"issn-type":[{"type":"print","value":"0167-7055"},{"type":"electronic","value":"1467-8659"}],"subject":[],"published":{"date-parts":[[2021,7]]},"assertion":[{"value":"2021-07-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}