{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T20:58:59Z","timestamp":1777928339503,"version":"3.51.4"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,7,11]],"date-time":"2016-07-11T00:00:00Z","timestamp":1468195200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["CapReal 335545"],"award-info":[{"award-number":["CapReal 335545"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2016,7,11]]},"abstract":"<jats:p>\n            Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational\n            <jats:italic>\u2113<\/jats:italic>\n            <jats:sub>2<\/jats:sub>\n            -\n            <jats:italic>\u2113<\/jats:italic>\n            <jats:sub>\n              <jats:italic>p<\/jats:italic>\n            <\/jats:sub>\n            -optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime\n            <jats:italic>and<\/jats:italic>\n            result quality -- even without user guidance such as scribbles.\n          <\/jats:p>","DOI":"10.1145\/2897824.2925907","type":"journal-article","created":{"date-parts":[[2016,7,11]],"date-time":"2016-07-11T16:04:33Z","timestamp":1468253073000},"page":"1-14","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":69,"title":["Live intrinsic video"],"prefix":"10.1145","volume":"35","author":[{"given":"Abhimitra","family":"Meka","sequence":"first","affiliation":[{"name":"Max Planck Institute for Informatics"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Zollh\u00f6fer","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Informatics"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Richardt","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Informatics and Intel Visual Computing Institute"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christian","family":"Theobalt","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Informatics"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,7,11]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.10"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2377712"},{"key":"e_1_2_2_3_1","volume-title":"Tech. Rep. 157","author":"Barrow H. G.","year":"1978"},{"key":"e_1_2_2_4_1","unstructured":"Bell M. and Freeman W. T. 2001. Learning local evidence for shading and reflection. In CVPR.  Bell M. and Freeman W. T. 2001. Learning local evidence for shading and reflection. In CVPR ."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601206"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766946"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661253"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818107"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1618452.1618476"},{"key":"e_1_2_2_10_1","volume-title":"J. W","author":"Chang J.","year":"2014"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.37"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2756549"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03137.x"},{"key":"e_1_2_2_14_1","unstructured":"Gehler P. V. Rother C. Kiefel M. Zhang L. and Sch\u00f6lkopf B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In NIPS.  Gehler P. V. Rother C. Kiefel M. Zhang L. and Sch\u00f6lkopf B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In NIPS ."},{"key":"e_1_2_2_15_1","doi-asserted-by":"crossref","unstructured":"Grosse R. Johnson M. K. Adelson E. H. and Freeman W. T. 2009. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV.  Grosse R. Johnson M. K. Adelson E. H. and Freeman W. T. 2009. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV .","DOI":"10.1109\/ICCV.2009.5459428"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.99"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.325"},{"key":"e_1_2_2_18_1","doi-asserted-by":"crossref","unstructured":"Holland P. W. and Welsch R. E. 1977. Robust regression using iteratively reweighted least-squares. Communications in Statistics -- Theory and Methods 6 9 (September) 813--827.  Holland P. W. and Welsch R. E. 1977. Robust regression using iteratively reweighted least-squares. Communications in Statistics -- Theory and Methods 6 9 (September) 813--827.","DOI":"10.1080\/03610927708827533"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/0146-664X(74)90022-7"},{"key":"e_1_2_2_20_1","doi-asserted-by":"crossref","unstructured":"Jiang X. Schofield A. J. and Wyatt J. L. 2010. Correlation-based intrinsic image extraction from a single image. In ECCV.   Jiang X. Schofield A. J. and Wyatt J. L. 2010. Correlation-based intrinsic image extraction from a single image. In ECCV .","DOI":"10.1007\/978-3-642-15561-1_5"},{"key":"e_1_2_2_21_1","doi-asserted-by":"crossref","unstructured":"Joshi N. Zitnick C. Szeliski R. and Kriegman D. 2009. Image deblurring and denoising using color priors. In CVPR.  Joshi N. Zitnick C. Szeliski R. and Kriegman D. 2009. Image deblurring and denoising using color priors. In CVPR .","DOI":"10.1109\/CVPR.2009.5206802"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISMAR.2007.4538852"},{"key":"e_1_2_2_23_1","doi-asserted-by":"crossref","unstructured":"Kong N. Gehler P. V. and Black M. J. 2014. Intrinsic video. In ECCV.  Kong N. Gehler P. V. and Black M. J. 2014. Intrinsic video. In ECCV .","DOI":"10.1007\/978-3-319-10605-2_24"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.57"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2366145.2366221"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.112"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1364\/JOSA.61.000001"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33783-3_24"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.1106"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276464"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.346"},{"key":"e_1_2_2_32_1","volume-title":"-Y","author":"Matsushita Y.","year":"2004"},{"key":"e_1_2_2_33_1","doi-asserted-by":"crossref","unstructured":"Shen L. and Yeo C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In CVPR.  Shen L. and Yeo C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In CVPR .","DOI":"10.1109\/CVPR.2011.5995738"},{"key":"e_1_2_2_34_1","unstructured":"Shen L. Tan P. and Lin S. 2008. Intrinsic image decomposition with non-local texture cues. In CVPR.  Shen L. Tan P. and Lin S. 2008. Intrinsic image decomposition with non-local texture cues. In CVPR ."},{"key":"e_1_2_2_35_1","doi-asserted-by":"crossref","unstructured":"Shen J. Yang X. Jia Y. and Li X. 2011. Intrinsic images using optimization. In CVPR.  Shen J. Yang X. Jia Y. and Li X. 2011. Intrinsic images using optimization. In CVPR .","DOI":"10.1109\/CVPR.2011.5995507"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2014.02.134"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2005.185"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03227.x"},{"key":"e_1_2_2_39_1","unstructured":"Weiss Y. 2001. Deriving intrinsic images from image sequences. In ICCV.  Weiss Y. 2001. Deriving intrinsic images from image sequences. In ICCV ."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141911.1142018"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661232"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601135"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.77"},{"key":"e_1_2_2_44_1","volume-title":"Generalized Schwarz Alternating Procedure for Domain Decomposition","author":"Zhao H. K."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.396"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601165"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766887"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.52"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897824.2925907","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2897824.2925907","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:55:03Z","timestamp":1750222503000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897824.2925907"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,11]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,11]]}},"alternative-id":["10.1145\/2897824.2925907"],"URL":"https:\/\/doi.org\/10.1145\/2897824.2925907","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,7,11]]},"assertion":[{"value":"2016-07-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}