{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T19:40:51Z","timestamp":1783107651017,"version":"3.54.6"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T00:00:00Z","timestamp":1669766400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61932003"],"award-info":[{"award-number":["61932003"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:p>Traditional differentiable rendering approaches are usually hard to converge in inverse rendering optimizations, especially when initial and target object locations are not so close. Inspired by Lagrangian fluid simulation, we present a novel differentiable rendering method to address this problem. We associate each screen-space pixel with the visible 3D geometric point covered by the center of the pixel and compute derivatives on geometric points rather than on pixels. We refer to the associated geometric points as point proxies of pixels. For each point proxy, we compute its 5D RGBXY derivatives which measures how its 3D RGB color and 2D projected screen-space position change with respect to scene parameters. Furthermore, in order to capture global and long-range object motions, we utilize optimal transport based pixel matching to design a more sophisticated loss function. We have conducted experiments to evaluate the effectiveness of our proposed method on various inverse rendering applications and have demonstrated superior convergence behavior compared to state-of-the-art baselines.<\/jats:p>","DOI":"10.1145\/3550454.3555479","type":"journal-article","created":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T21:19:07Z","timestamp":1669843147000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Differentiable Rendering Using RGBXY Derivatives and Optimal Transport"],"prefix":"10.1145","volume":"41","author":[{"given":"Jiankai","family":"Xing","sequence":"first","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fujun","family":"Luan","sequence":"additional","affiliation":[{"name":"Adobe Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ling-Qi","family":"Yan","sequence":"additional","affiliation":[{"name":"University of California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuejun","family":"Hu","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Houde","family":"Qian","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kun","family":"Xu","sequence":"additional","affiliation":[{"name":"Tsinghua University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,11,30]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417833"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323021"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925918"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2024156.2024192"},{"key":"e_1_2_2_5_1","unstructured":"Robert Bridson. 2008. Fluid Simulation for Computer Graphics. A K Peters\/CRC Press."},{"key":"e_1_2_2_6_1","volume-title":"Physics-Based Inverse Rendering using Combined Implicit and Explicit Geometries. arXiv preprint arXiv:2205.01242","author":"Cai Guangyan","year":"2022","unstructured":"Guangyan Cai, Kai Yan, Zhao Dong, Ioannis Gkioulekas, and Shuang Zhao. 2022. Physics-Based Inverse Rendering using Combined Implicit and Explicit Geometries. arXiv preprint arXiv:2205.01242 (2022)."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCP48838.2020.9105209"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAES.2016.140952"},{"key":"e_1_2_2_9_1","volume-title":"Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems 26","author":"Cuturi Marco","year":"2013","unstructured":"Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems 26 (2013)."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528233.3530714"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/274976.274981"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/383259.383260"},{"key":"e_1_2_2_13_1","volume-title":"The 22nd International Conference on Artificial Intelligence and Statistics. 2681--2690","author":"Feydy Jean","year":"2019","unstructured":"Jean Feydy, Thibault S\u00e9journ\u00e9, Fran\u00e7ois-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyr\u00e9. 2019. Interpolating between Optimal Transport and MMD using Sinkhorn Divergences. In The 22nd International Conference on Artificial Intelligence and Statistics. 2681--2690."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01075"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323042"},{"key":"e_1_2_2_16_1","volume-title":"MaterialGAN: reflectance capture using a generative SVBRDF model. arXiv preprint arXiv:2010.00114","author":"Guo Yu","year":"2020","unstructured":"Yu Guo, Cameron Smith, Milo\u0161 Ha\u0161an, Kalyan Sunkavalli, and Shuang Zhao. 2020. MaterialGAN: reflectance capture using a generative SVBRDF model. arXiv preprint arXiv:2010.00114 (2020)."},{"key":"e_1_2_2_17_1","volume-title":"Eurographics Symposium on Rendering.","author":"Hasselgren Jon","year":"2021","unstructured":"Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, and Samuli Laine. 2021. Appearance-Driven Automatic 3D Model Simplification. In Eurographics Symposium on Rendering."},{"key":"e_1_2_2_18_1","volume-title":"Jittor: a novel deep learning framework with meta-operators and unified graph execution. Science China Information Sciences 63, 222103","author":"Hu Shi-Min","year":"2020","unstructured":"Shi-Min Hu, Dun Liang, Guo-Ye Yang, Guo-Wei Yang, and Wen-Yang Zhou. 2020. Jittor: a novel deep learning framework with meta-operators and unified graph execution. Science China Information Sciences 63, 222103 (2020), 1--222103. https:\/\/github.com\/Jittor\/jrender"},{"key":"e_1_2_2_19_1","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR). http:\/\/lmb.informatik.unifreiburg.de\/\/Publications\/2017\/IMKDB17","author":"Ilg E.","unstructured":"E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. 2017. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). http:\/\/lmb.informatik.unifreiburg.de\/\/Publications\/2017\/IMKDB17"},{"key":"e_1_2_2_20_1","first-page":"199","article-title":"On the translocation of masses","volume":"37","author":"Kantorovich Leonid V","year":"1942","unstructured":"Leonid V Kantorovich. 1942. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), Vol. 37. 199--201.","journal-title":"Dokl. Akad. Nauk. USSR (NS)"},{"key":"e_1_2_2_21_1","volume-title":"Differentiable rendering: A survey. arXiv preprint arXiv:2006.12057","author":"Kato Hiroharu","year":"2020","unstructured":"Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. 2020a. Differentiable rendering: A survey. arXiv preprint arXiv:2006.12057 (2020)."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2006.12057"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00411"},{"key":"e_1_2_2_24_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_25_1","volume-title":"The Hungarian method for the assignment problem. Naval Res. Logist. Quart","author":"Kuhn H. W.","year":"1955","unstructured":"H. W. Kuhn and Bryn Yaw. 1955. The Hungarian method for the assignment problem. Naval Res. Logist. Quart (1955), 83--97."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417861"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.2312\/egst.20141042"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275055"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00780"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818013"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10584-0_11"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356510"},{"key":"e_1_2_2_33_1","volume-title":"Computer Graphics Forum","author":"Luan Fujun","unstructured":"Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. 2021. Unified shape and svbrdf recovery using differentiable monte carlo rendering. In Computer Graphics Forum, Vol. 40. Wiley Online Library, 101--113."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_24"},{"key":"e_1_2_2_35_1","unstructured":"Gaspard Monge. 1781. M\u00e9moire sur la th\u00e9orie des d\u00e9blais et des remblais. Histoire de l'Acad\u00e9mie Royale des Sciences de Paris (1781)."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.5555\/846276.846298"},{"key":"e_1_2_2_37_1","volume-title":"Extracting Triangular 3D Models, Materials, and Lighting From Images. arXiv preprint arXiv:2111.12503","author":"Munkberg Jacob","year":"2021","unstructured":"Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas M\u00fcller, and Sanja Fidler. 2021. Extracting Triangular 3D Models, Materials, and Lighting From Images. arXiv preprint arXiv:2111.12503 (2021)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480501"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392395"},{"key":"e_1_2_2_40_1","doi-asserted-by":"crossref","unstructured":"Gabriel Peyr\u00e9 Marco Cuturi et al. 2019. Computational optimal transport: With applications to data science. Foundations and Trends\u00ae in Machine Learning 11 5--6 (2019) 355--607.","DOI":"10.1561\/2200000073"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-8659.00687"},{"key":"e_1_2_2_42_1","volume-title":"Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501","author":"Ravi Nikhila","year":"2020","unstructured":"Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. 2020. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.08501 (2020)."},{"key":"e_1_2_2_43_1","unstructured":"Liang Shi Beichen Li Milo\u0161 Ha\u0161an Kalyan Sunkavalli Tamy Boubekeur Radomir Mech and Wojciech Matusik. 2020. MATch: Differentiable material graphs for procedural material capture. (2020)."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766963"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/311535.311548"},{"key":"e_1_2_2_46_1","volume-title":"Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR. https:\/\/arxiv.org\/abs\/2203","author":"Truong Prune","year":"2022","unstructured":"Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. 2022. Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR. https:\/\/arxiv.org\/abs\/2203.04279"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530139"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACSSC.2003.1292216"},{"key":"e_1_2_2_49_1","volume-title":"Path-space differentiable rendering. ACM transactions on graphics 39, 4","author":"Zhang Cheng","year":"2020","unstructured":"Cheng Zhang, Bailey Miller, Kan Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-space differentiable rendering. ACM transactions on graphics 39, 4 (2020)."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478513.3480500"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"},{"key":"e_1_2_2_52_1","volume-title":"CVPR 2021 Tutorial on Physics-Based Differentiable Rendering. https:\/\/www.diff-render.org\/.","author":"Zhao Shuang","year":"2021","unstructured":"Shuang Zhao, Ioannis Gkioulekas, and Sai Bangaru. 2021. CVPR 2021 Tutorial on Physics-Based Differentiable Rendering. https:\/\/www.diff-render.org\/."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550454.3555479","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3550454.3555479","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:51:43Z","timestamp":1750182703000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3550454.3555479"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,30]]},"references-count":52,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["10.1145\/3550454.3555479"],"URL":"https:\/\/doi.org\/10.1145\/3550454.3555479","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,30]]},"assertion":[{"value":"2022-11-30","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}