{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T16:45:30Z","timestamp":1793205930322,"version":"4.5.0"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T00:00:00Z","timestamp":1597190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2020,8,31]]},"abstract":"<jats:p>\n            Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and dark parts of the face are frequently determined by the constraints of the environment rather than by the photographer. Professionals address this issue by adding light shaping tools such as scrims, bounce cards, and flashes. In this paper, we present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks---one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio. To train our first network we construct a dataset of real-world portraits wherein synthetic foreign shadows are rendered onto the face, and we show that our network learns to remove those unwanted shadows. To train our second network we use a dataset of Light Stage scans of human subjects to construct input\/output pairs of input images harshly lit by a small light source, and variably softened and fill-lit output images of each face. We propose a way to explicitly encode facial symmetry and show that our dataset and training procedure enable the model to generalize to images taken in the wild. Together, these networks enable the realistic and aesthetically pleasing enhancement of shadows and lights in real-world portrait images.\n            <jats:sup>1<\/jats:sup>\n          <\/jats:p>","DOI":"10.1145\/3386569.3392390","type":"journal-article","created":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T11:44:27Z","timestamp":1597232667000},"update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":91,"title":["Portrait shadow manipulation"],"prefix":"10.1145","volume":"39","author":[{"given":"Xuaner (Cecilia)","family":"Zhang","sequence":"first","affiliation":[{"name":"University of California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan T.","family":"Barron","sequence":"additional","affiliation":[{"name":"Google Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun-Ta","family":"Tsai","sequence":"additional","affiliation":[{"name":"Google Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohit","family":"Pandey","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiuming","family":"Zhang","sequence":"additional","affiliation":[{"name":"MIT"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ren","family":"Ng","sequence":"additional","affiliation":[{"name":"University of California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David E.","family":"Jacobs","sequence":"additional","affiliation":[{"name":"Google Research"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,8,12]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Shadow removal using intensity surfaces and texture anchor points","author":"Arbel Eli","year":"2010","unstructured":"Eli Arbel and Hagit Hel-Or. 2010. Shadow removal using intensity surfaces and texture anchor points. IEEE TPAMI (2010)."},{"key":"e_1_2_2_2_1","doi-asserted-by":"crossref","unstructured":"Masashi Baba and Naoki Asada. 2003. Shadow Removal from a Real Picture. In SIGGRAPH.","DOI":"10.1145\/965400.965488"},{"key":"e_1_2_2_3_1","volume-title":"Barron and Jitendra Malik","author":"Jonathan","year":"2015","unstructured":"Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. TPAMI (2015)."},{"key":"e_1_2_2_4_1","doi-asserted-by":"crossref","unstructured":"Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In ICCV.","DOI":"10.1109\/ICCV.2017.168"},{"key":"e_1_2_2_5_1","volume-title":"Fsrnet: End-to-end learning face super-resolution with facial priors. In CVPR.","author":"Chen Yu","year":"2018","unstructured":"Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. 2018. Fsrnet: End-to-end learning face super-resolution with facial priors. In CVPR."},{"key":"e_1_2_2_6_1","unstructured":"Xiaodong Cun Chi-Man Pun and Cheng Shi. 2020. Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN. AAAI."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/344779.344855"},{"key":"e_1_2_2_8_1","unstructured":"Emily L Denton Soumith Chintala Rob Fergus et al. 2015. Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks. In NIPS."},{"key":"e_1_2_2_9_1","volume-title":"ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal. In ICCV.","author":"Ding Bin","year":"2019","unstructured":"Bin Ding, Chengjiang Long, Ling Zhang, and Chunxia Xiao. 2019. ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal. In ICCV."},{"key":"e_1_2_2_10_1","doi-asserted-by":"crossref","unstructured":"Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin. (2006).","DOI":"10.1145\/1179849.1180033"},{"key":"e_1_2_2_11_1","unstructured":"Randima Fernando. 2004. GPU Gems: Programming Techniques Tips and Tricks for Real-Time Graphics. Pearson Higher Education."},{"key":"e_1_2_2_12_1","volume-title":"Entropy minimization for shadow removal. IJCV","author":"Finlayson Graham D","year":"2009","unstructured":"Graham D Finlayson, Mark S Drew, and Cheng Lu. 2009. Entropy minimization for shadow removal. IJCV (2009)."},{"key":"e_1_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Graham D Finlayson Steven D Hordley and Mark S Drew. 2002. Removing Shadows from Images. In ECCV.","DOI":"10.1007\/3-540-47979-1_55"},{"key":"e_1_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Damien Fourure R\u00e9mi Emonet Elisa Fromont Damien Muselet Alain Tremeau and Christian Wolf. 2017. Residual Conv-Deconv Grid Network for Semantic Segmentation. In BMVC.","DOI":"10.5244\/C.31.181"},{"key":"e_1_2_2_15_1","doi-asserted-by":"crossref","unstructured":"Micha\u00ebl Gharbi Jiawen Chen Jonathan T. Barron Samuel W Hasinoff and Fr\u00e9do Durand. 2017. Deep bilateral learning for real-time image enhancement. In SIGGRAPH.","DOI":"10.1145\/3072959.3073592"},{"key":"e_1_2_2_16_1","unstructured":"Christopher Grey. 2014. Master lighting guide for portrait photographers. Amherst Media."},{"key":"e_1_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Roger Grosse Micah K Johnson Edward H Adelson and William T Freeman. 2009. Ground Truth Dataset and Baseline Evaluations for Intrinsic Image Algorithms. In ICCV.","DOI":"10.1109\/ICCV.2009.5459428"},{"key":"e_1_2_2_18_1","volume-title":"Brostow","author":"Gryka Maciej","year":"2015","unstructured":"Maciej Gryka, Michael Terry, and Gabriel J. Brostow. 2015. Learning to Remove Soft Shadows. ACM TOG (2015)."},{"key":"e_1_2_2_19_1","volume-title":"Paired regions for shadow detection and removal. TPAMI","author":"Guo Ruiqi","year":"2012","unstructured":"Ruiqi Guo, Qieyun Dai, and Derek Hoiem. 2012. Paired regions for shadow detection and removal. TPAMI (2012)."},{"key":"e_1_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Pat Hanrahan and Wolfgang Krueger. 1993. Reflection from Layered Surfaces Due to Subsurface Scattering. In SIGGRAPH.","DOI":"10.1145\/166117.166139"},{"key":"e_1_2_2_21_1","unstructured":"Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR."},{"key":"e_1_2_2_22_1","volume-title":"Determining lightness from an image. Computer Graphics and Image Processing","author":"Horn Berthold K. P.","year":"1974","unstructured":"Berthold K. P. Horn. 1974. Determining lightness from an image. Computer Graphics and Image Processing (1974)."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2919616"},{"key":"e_1_2_2_24_1","unstructured":"Xiaowei Hu Lei Zhu Chi-Wing Fu Jing Qin and Pheng-Ann Heng. 2018. Direction-Aware Spatial Context Features for Shadow Detection. In CVPR."},{"key":"e_1_2_2_25_1","unstructured":"Jinggang Huang Ann B Lee and David Mumford. 2000. Statistics of range images. In CVPR."},{"key":"e_1_2_2_26_1","doi-asserted-by":"crossref","unstructured":"Henrik Wann Jensen Stephen R Marschner Marc Levoy and Pat Hanrahan. 2001. A Practical Model for Subsurface Light Transport. In SIGGRAPH.","DOI":"10.1145\/383259.383319"},{"key":"e_1_2_2_27_1","unstructured":"Yury Kartynnik Artsiom Ablavatski Ivan Grishchenko and Matthias Grundmann. 2019. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. (2019)."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03225.x"},{"key":"e_1_2_2_29_1","volume-title":"Automatic Shadow Detection and Removal from a Single Image","author":"Khan Salman H","year":"2015","unstructured":"Salman H Khan, Mohammed Bennamoun, Ferdous Sohel, and Roberto Togneri. 2015. Automatic Shadow Detection and Removal from a Single Image. IEEE TPAMI (2015)."},{"key":"e_1_2_2_30_1","volume-title":"Kingma and Jimmy Ba","author":"Diederik","year":"2015","unstructured":"Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR."},{"key":"e_1_2_2_31_1","doi-asserted-by":"crossref","unstructured":"Aravind Krishnaswamy and Gladimir VG Baranoski. 2004. A biophysically-based spectral model of light interaction with human skin. In Computer Graphics Forum.","DOI":"10.1111\/j.1467-8659.2004.00764.x"},{"key":"e_1_2_2_32_1","unstructured":"Vuong Le Jonathan Brandt Zhe Lin Lubomir Bourdev and Thomas S Huang. 2012. Interactive facial feature localization. In ECCV."},{"key":"e_1_2_2_33_1","unstructured":"Li-Qian Ma Jue Wang Eli Shechtman Kalyan Sunkavalli and Shi-Min Hu. 2016. Appearance harmonization for single image shadow removal. In Computer Graphics Forum."},{"key":"e_1_2_2_34_1","doi-asserted-by":"crossref","unstructured":"Simon Niklaus and Feng Liu. 2018. Context-aware synthesis for video frame interpolation. In CVPR.","DOI":"10.1109\/CVPR.2018.00183"},{"key":"e_1_2_2_35_1","volume-title":"3D Ken Burns effect from a single image. ACM TOG","author":"Niklaus Simon","year":"2019","unstructured":"Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 2019. 3D Ken Burns effect from a single image. ACM TOG (2019)."},{"key":"e_1_2_2_36_1","volume-title":"Lau","author":"Qu Liangqiong","year":"2017","unstructured":"Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson W. H. Lau. 2017. DeshadowNet: A Multi-Context Embedding Deep Network for Shadow Removal. In CVPR."},{"key":"e_1_2_2_37_1","doi-asserted-by":"crossref","unstructured":"Ravi Ramamoorthi and Pat Hanrahan. 2001. A Signal-Processing Framework for Inverse Rendering. In SIGGRAPH.","DOI":"10.1145\/383259.383271"},{"key":"e_1_2_2_38_1","volume-title":"The Influence of Cast Shadows on Visual Search. Perception","author":"Rensink Ronald A","year":"2004","unstructured":"Ronald A Rensink and Patrick Cavanagh. 2004. The Influence of Cast Shadows on Visual Search. Perception (2004)."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2003.1182093"},{"key":"e_1_2_2_40_1","volume-title":"Jacobs","author":"Sengupta Soumyadip","year":"2018","unstructured":"Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning Shape, Refectance and Illuminance of Faces in the Wild. In CVPR."},{"key":"e_1_2_2_41_1","volume-title":"The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations","author":"Shashua Amnon","year":"2001","unstructured":"Amnon Shashua and Tammy Riklin-Raviv. 2001. The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations. IEEE TPAMI (2001)."},{"key":"e_1_2_2_42_1","volume-title":"Style Transfer for Headshot Portraits. SIGGRAPH","author":"Shih YiChang","year":"2014","unstructured":"YiChang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Fr\u00e9do Durand. 2014. Style Transfer for Headshot Portraits. SIGGRAPH (2014)."},{"key":"e_1_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Yael Shor and Dani Lischinski. 2008. The shadow meets the mask: Pyramid-based shadow removal. In Computer Graphics Forum.","DOI":"10.1111\/j.1467-8659.2008.01155.x"},{"key":"e_1_2_2_44_1","unstructured":"Zhixin Shu Sunil Hadap Eli Shechtman Kalyan Sunkavalli Sylvain Paris and Dimitris Samaras. 2017. Portrait lighting transfer using a mass transport approach. In SIGGRAPH."},{"key":"e_1_2_2_45_1","volume-title":"Single Image Portrait Relighting. SIGGRAPH","author":"Sun Tiancheng","year":"2019","unstructured":"Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E. Debevec, and Ravi Ramamoorthi. 2019. Single Image Portrait Relighting. SIGGRAPH (2019)."},{"key":"e_1_2_2_46_1","unstructured":"Dmitry Ulyanov Andrea Vedaldi and Victor Lempitsky. 2018. Deep image prior. In CVPR."},{"key":"e_1_2_2_47_1","doi-asserted-by":"crossref","unstructured":"Jifeng Wang Xiang Li and Jian Yang. 2018. Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal. In CVPR.","DOI":"10.1109\/CVPR.2018.00192"},{"key":"e_1_2_2_48_1","volume-title":"Simoncelli","author":"Wang Zhou","year":"2004","unstructured":"Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE TIP (2004)."},{"key":"e_1_2_2_49_1","volume-title":"Natural shadow matting. ACM TOG","author":"Wu Tai-Pang","year":"2007","unstructured":"Tai-Pang Wu, Chi-Keung Tang, Michael S Brown, and Heung-Yeung Shum. 2007. Natural shadow matting. ACM TOG (2007)."},{"key":"e_1_2_2_50_1","volume-title":"Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do.","author":"Yeh Raymond A","year":"2017","unstructured":"Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. 2017. Semantic image inpainting with deep generative models. In CVPR."},{"key":"e_1_2_2_51_1","volume-title":"Effective Shadow Removal Via Multi-Scale Image Decomposition. The Visual Computer","author":"Zhang Ling","year":"2019","unstructured":"Ling Zhang, Qingan Yan, Yao Zhu, Xiaolong Zhang, and Chunxia Xiao. 2019b. Effective Shadow Removal Via Multi-Scale Image Decomposition. The Visual Computer (2019)."},{"key":"e_1_2_2_52_1","doi-asserted-by":"crossref","unstructured":"Richard Zhang Phillip Isola Alexei A. Efros Eli Shechtman and Oliver Wang. 2018a. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323015"},{"key":"e_1_2_2_54_1","doi-asserted-by":"crossref","unstructured":"Xuaner Zhang Ren Ng and Qifeng Chen. 2018b. Single Image Reflection Removal with Perceptual Losses. In CVPR.","DOI":"10.1109\/CVPR.2018.00503"},{"key":"e_1_2_2_55_1","doi-asserted-by":"crossref","unstructured":"Quanlong Zheng Xiaotian Qiao Ying Cao and Rynson WH Lau. 2019. Distraction-aware shadow detection. In CVPR. 5167--5176.","DOI":"10.1109\/CVPR.2019.00531"},{"key":"e_1_2_2_56_1","doi-asserted-by":"crossref","unstructured":"Hao Zhou Sunil Hadap Kalyan Sunkavalli and David W Jacobs. 2019. Deep Single-Image Portrait Relighting. In ICCV.","DOI":"10.1109\/ICCV.2019.00729"},{"key":"e_1_2_2_57_1","doi-asserted-by":"crossref","unstructured":"Lei Zhu Zijun Deng Xiaowei Hu Chi-Wing Fu Xuemiao Xu Jing Qin and Pheng-Ann Heng. 2018. Bidirectional Feature Pyramid Network with Recurrent Attention Residual Modules for Shadow Detection. In ECCV.","DOI":"10.1007\/978-3-030-01231-1_8"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392390","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3386569.3392390","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T05:37:44Z","timestamp":1750829864000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392390"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,12]]},"references-count":57,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,8,31]]}},"alternative-id":["10.1145\/3386569.3392390"],"URL":"https:\/\/doi.org\/10.1145\/3386569.3392390","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,12]]},"assertion":[{"value":"2020-08-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}