{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T18:56:08Z","timestamp":1776884168589,"version":"3.51.2"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,7,1]],"date-time":"2022-07-01T00:00:00Z","timestamp":1656633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1900849, 2008464, 1730147"],"award-info":[{"award-number":["1900849, 2008464, 1730147"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000879","name":"Alfred P. Sloan Foundation","doi-asserted-by":"publisher","award":["FG202013153"],"award-info":[{"award-number":["FG202013153"]}],"id":[{"id":"10.13039\/100000879","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>Reconstructing and designing media with continuously-varying refractive index fields remains a challenging problem in computer graphics. A core difficulty in trying to tackle this inverse problem is that light travels inside such media along curves, rather than straight lines. Existing techniques for this problem make strong assumptions on the shape of the ray inside the medium, and thus limit themselves to media where the ray deflection is relatively small. More recently, differentiable rendering techniques have relaxed this limitation, by making it possible to differentiably simulate curved light paths. However, the automatic differentiation algorithms underlying these techniques use large amounts of memory, restricting existing differentiable rendering techniques to relatively small media and low spatial resolutions.<\/jats:p>\n          <jats:p>We present a method for optimizing refractive index fields that both accounts for curved light paths and has a small, constant memory footprint. We use the adjoint state method to derive a set of equations for computing derivatives with respect to the refractive index field of optimization objectives that are subject to nonlinear ray tracing constraints. We additionally introduce discretization schemes to numerically evaluate these equations, without the need to store nonlinear ray trajectories in memory, significantly reducing the memory requirements of our algorithm. We use our technique to optimize high-resolution refractive index fields for a variety of applications, including creating different types of displays (multiview, lightfield, caustic), designing gradient-index optics, and reconstructing gas flows.<\/jats:p>","DOI":"10.1145\/3528223.3530077","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T21:06:27Z","timestamp":1658523987000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Adjoint nonlinear ray tracing"],"prefix":"10.1145","volume":"41","author":[{"given":"Arjun","family":"Teh","sequence":"first","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"O'Toole","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ioannis","family":"Gkioulekas","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,22]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Refractive radiative transfer equation. ACM TOG","author":"Ament Marco","year":"2014","unstructured":"Marco Ament, Christoph Bergmann, and Daniel Weiskopf. 2014. Refractive radiative transfer equation. ACM TOG (2014)."},{"key":"e_1_2_2_2_1","volume-title":"Derek Bradley, Marcus Magnor, and Hans-Peter Seidel.","author":"Atcheson Bradley","year":"2008","unstructured":"Bradley Atcheson, Ivo Ihrke, Wolfgang Heidrich, Art Tevs, Derek Bradley, Marcus Magnor, and Hans-Peter Seidel. 2008. Time-resolved 3d capture of non-stationary gas flows. ACM TOG (2008)."},{"key":"e_1_2_2_3_1","volume-title":"Werner","author":"Balasubramanian Manushanker","year":"2020","unstructured":"Manushanker Balasubramanian, Sawyer D. Campbell, and Douglas H. Werner. 2020. Highly-efficient GRIN Lens Optimization Through Differential Ray Tracing. IEEE ISAP (2020)."},{"key":"e_1_2_2_4_1","unstructured":"Max Born and Emil Wolf. 2013. Principles of optics: electromagnetic theory of propagation interference and diffraction of light."},{"key":"e_1_2_2_5_1","volume-title":"Julien Bloch, Ninh Do, Minyoung Huh, Dongjin Seo, Jillian Iafrati, Vikaas S Sohal, Mohammad-Reza Alam, and Michel M Maharbiz.","author":"Chamanzar Maysamreza","year":"2019","unstructured":"Maysamreza Chamanzar, Matteo Giuseppe Scopelliti, Julien Bloch, Ninh Do, Minyoung Huh, Dongjin Seo, Jillian Iafrati, Vikaas S Sohal, Mohammad-Reza Alam, and Michel M Maharbiz. 2019. Ultrasonic sculpting of virtual optical waveguides in tissue. Nature Communications (2019)."},{"key":"e_1_2_2_6_1","volume-title":"Identification of Functional Parameters in Partial Differential Equations. Joint Automatic Control Conference.","author":"Chavent Guy","year":"1974","unstructured":"Guy Chavent. 1974. Identification of Functional Parameters in Partial Differential Equations. Joint Automatic Control Conference."},{"key":"e_1_2_2_7_1","volume-title":"Neural Ordinary Differential Equations. NeurIPS","author":"Chen Ricky T. Q.","year":"2018","unstructured":"Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural Ordinary Differential Equations. NeurIPS (2018)."},{"key":"e_1_2_2_8_1","volume-title":"Evaluation of some GRIN fiber parameters and the associated fraction mode loss due to mechanically induced optical anisotropy. Applied optics","author":"El-Diasty Fouad","year":"2003","unstructured":"Fouad El-Diasty. 2003. Evaluation of some GRIN fiber parameters and the associated fraction mode loss due to mechanically induced optical anisotropy. Applied optics (2003)."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417766"},{"key":"e_1_2_2_10_1","volume-title":"The reversible residual network: Backpropagation without storing activations. NeurIPS","author":"Gomez Aidan N","year":"2017","unstructured":"Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017. The reversible residual network: Backpropagation without storing activations. NeurIPS (2017)."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898717761"},{"key":"e_1_2_2_12_1","volume-title":"Nonlinear ray tracing: Visualizing strange worlds. The Visual Computer","author":"Gr\u00f6ller Eduard","year":"1995","unstructured":"Eduard Gr\u00f6ller. 1995. Nonlinear ray tracing: Visualizing strange worlds. The Visual Computer (1995)."},{"key":"e_1_2_2_13_1","volume-title":"Global illumination in inhomogeneous media based on curved photon mapping. Visualization, Imaging, and Image Processing","author":"Gutierrez Diego","year":"2003","unstructured":"Diego Gutierrez, A Mu\u00f1oz, F Seron, E Jimenez, Mar\u00eda de Luna, and Edificio Ada Byron. 2003. Global illumination in inhomogeneous media based on curved photon mapping. Visualization, Imaging, and Image Processing (2003)."},{"key":"e_1_2_2_14_1","volume-title":"Geometric numerical integration","author":"Hairer Ernst","unstructured":"Ernst Hairer, Christian Lubich, and Gerhard Wanner. 2006. Geometric numerical integration. Springer-Verlag, Berlin,."},{"key":"e_1_2_2_15_1","volume-title":"Optimization with PDE constraints","author":"Hinze Michael","unstructured":"Michael Hinze, Ren\u00e9 Pinnau, Michael Ulbrich, and Stefan Ulbrich. 2008. Optimization with PDE constraints. Springer."},{"key":"e_1_2_2_16_1","volume-title":"Walker","author":"Hisatomi Makiko","year":"2005","unstructured":"Makiko Hisatomi, Michael C. Parker, and Stuart D. Walker. 2005. Comparison of zoned microstructure fiber geometries for low-dispersion waveguiding. Journal of Lightwave Technology (2005)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Ivo Ihrke. 2007. Reconstruction and rendering of time-varying natural phenomena. (2007).","DOI":"10.1145\/1275808.1276451"},{"key":"e_1_2_2_18_1","volume-title":"Christian Theobalt, Marcus Magnor, and Hans-Peter Seidel.","author":"Ihrke Ivo","year":"2007","unstructured":"Ivo Ihrke, Gernot Ziegler, Art Tevs, Christian Theobalt, Marcus Magnor, and Hans-Peter Seidel. 2007. Eikonal rendering: Efficient light transport in refractive objects. ACM TOG (2007)."},{"key":"e_1_2_2_19_1","unstructured":"Wenzel Jakob. 2019. Enoki: structured vectorization and differentiation on modern processor architectures. https:\/\/github.com\/mitsuba-renderer\/enoki."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2013.324"},{"key":"e_1_2_2_21_1","unstructured":"SeungYeon Kang Elena Dotsenko David Amrhein Christian Theriault and Craig B Arnold. 2018. Ultra-high-speed variable focus optics for novel applications in advanced imaging. In Photonic Instrumentation Engineering V."},{"key":"e_1_2_2_22_1","volume-title":"Geometric, variational integrators for computer animation","author":"Kharevych Liliya","unstructured":"Liliya Kharevych, W Wei, Yiying Tong, Eva Kanso, Jerrold E Marsden, Peter Schr\u00f6der, and Matthieu Desbrun. 2006. Geometric, variational integrators for computer animation. Eurographics Association."},{"key":"e_1_2_2_23_1","volume-title":"Adam: A method for stochastic optimization. arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_24_1","volume-title":"Geometrical optics of inhomogeneous media","author":"Kravtsov Yu A","unstructured":"Yu A Kravtsov and Yu I Orlov. 1990. Geometrical optics of inhomogeneous media. Springer."},{"key":"e_1_2_2_25_1","doi-asserted-by":"crossref","unstructured":"Shingyu Leung Jianliang Qian et al. 2006. An adjoint state method for three-dimensional transmission traveltime tomography using first-arrivals. Communications in Mathematical Sciences (2006).","DOI":"10.21236\/ADA448160"},{"key":"e_1_2_2_26_1","volume-title":"Differentiable Monte Carlo ray tracing through edge sampling. ACM TOG","author":"Li Tzu-Mao","year":"2018","unstructured":"Tzu-Mao Li, Miika Aittala, Fr\u00e9do Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo ray tracing through edge sampling. ACM TOG (2018)."},{"key":"e_1_2_2_27_1","volume-title":"End-to-end learned single lens design using fast differentiable ray tracing. Optics Letters","author":"Li Zongling","year":"2021","unstructured":"Zongling Li, Qingyu Hou, Zhipeng Wang, Fanjiao Tan, Jin Liu, and Wei Zhang. 2021. End-to-end learned single lens design using fast differentiable ray tracing. Optics Letters (2021)."},{"key":"e_1_2_2_28_1","volume-title":"Mathematical Theory of Optics. Providence","author":"Luneberg Rudolf K","unstructured":"Rudolf K Luneberg. 1944. Mathematical Theory of Optics. Providence. Brown Univ. Press."},{"key":"e_1_2_2_29_1","volume-title":"Reversible recurrent neural networks. NeurIPS","author":"MacKay Matthew","year":"2018","unstructured":"Matthew MacKay, Paul Vicol, Jimmy Ba, and Roger B Grosse. 2018. Reversible recurrent neural networks. NeurIPS (2018)."},{"key":"e_1_2_2_30_1","volume-title":"Solutions of problems","author":"Maxwell James Clerk","year":"1854","unstructured":"James Clerk Maxwell. 1854. Solutions of problems. Cambridge Dublin Math. J. (1854)."},{"key":"e_1_2_2_31_1","volume-title":"Fluid control using the adjoint method. ACM TOG","author":"McNamara Antoine","year":"2004","unstructured":"Antoine McNamara, Adrien Treuille, Zoran Popovi\u0107, and Jos Stam. 2004. Fluid control using the adjoint method. ACM TOG (2004)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275056"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392406"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356498"},{"key":"e_1_2_2_35_1","volume-title":"Computer Graphics Forum","author":"Papas Marios","unstructured":"Marios Papas, Wojciech Jarosz, Wenzel Jakob, Szymon Rusinkiewicz, Wojciech Matusik, and Tim Weyrich. 2011. Goal-based caustics. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 503--511."},{"key":"e_1_2_2_36_1","volume-title":"PyTorch: An Imperative Style","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS (2019)."},{"key":"e_1_2_2_37_1","volume-title":"Matteo Giuseppe Scopelliti, Maysamreza Chamanzar, Srinivasa Narasimhan, and Ioannis Gkioulekas.","author":"Pediredla Adithya","year":"2020","unstructured":"Adithya Pediredla, Yasin Karimi Chalmiani, Matteo Giuseppe Scopelliti, Maysamreza Chamanzar, Srinivasa Narasimhan, and Ioannis Gkioulekas. 2020. Path tracing estimators for refractive radiative transfer. ACM TOG (2020)."},{"key":"e_1_2_2_38_1","volume-title":"A review of the adjoint-state method for computing the gradient of a functional with geophysical applications. Geophysical Journal International","author":"Plessix Rene-Edouard","year":"2006","unstructured":"Rene-Edouard Plessix. 2006. A review of the adjoint-state method for computing the gradient of a functional with geophysical applications. Geophysical Journal International (2006)."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1088\/0266-5611\/32\/8\/085009"},{"key":"e_1_2_2_40_1","volume-title":"High-contrast computational caustic design. ACM TOG","author":"Schwartzburg Yuliy","year":"2014","unstructured":"Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast computational caustic design. ACM TOG (2014)."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41377-019-0173-7"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1364\/OE.404986"},{"key":"e_1_2_2_43_1","series-title":"SIAM review","volume-title":"Fast marching methods","author":"Sethian James A","year":"1999","unstructured":"James A Sethian. 1999. Fast marching methods. SIAM review (1999)."},{"key":"e_1_2_2_44_1","volume-title":"Dhanwada Vizia Kumar, and Ajoy K. Ghatak","author":"Sharma Anurag","year":"1982","unstructured":"Anurag Sharma, Dhanwada Vizia Kumar, and Ajoy K. Ghatak. 1982. Tracing rays through graded-index media: a new method. Applied Optics (1982)."},{"key":"e_1_2_2_45_1","volume-title":"Variational surface cutting. ACM TOG","author":"Sharp Nicholas","year":"2018","unstructured":"Nicholas Sharp and Keenan Crane. 2018. Variational surface cutting. ACM TOG (2018)."},{"key":"e_1_2_2_46_1","volume-title":"Computing Light Transport Gradients using the Adjoint Method. arXiv:2006.15059","author":"Stam Jos","year":"2020","unstructured":"Jos Stam. 2020. Computing Light Transport Gradients using the Adjoint Method. arXiv:2006.15059 (2020)."},{"key":"e_1_2_2_47_1","doi-asserted-by":"crossref","unstructured":"Jos Stam and Eric Langu\u00e9nou. 1996. Ray tracing in non-constant media. In EGSR.","DOI":"10.1007\/978-3-7091-7484-5_23"},{"key":"e_1_2_2_48_1","volume-title":"End-to-End Complex Lens Design with Differentiable Ray Tracing. ACM TOG","author":"Sun Qilin","year":"2021","unstructured":"Qilin Sun, Congli Wang, Fu Qiang, Dun Xiong, and Heidrich Wolfgang. 2021. End-to-End Complex Lens Design with Differentiable Ray Tracing. ACM TOG (2021)."},{"key":"e_1_2_2_50_1","volume-title":"Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. ACM TOG","author":"Tseng Ethan","year":"2021","unstructured":"Ethan Tseng, Ali Mosleh, Fahim Mannan, Karl St-Arnaud, Avinash Sharma, Yifan Peng, Alexander Braun, Derek Nowrouzezahrai, Jean-Francois Lalonde, and Felix Heide. 2021. Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. ACM TOG (2021)."},{"key":"e_1_2_2_51_1","volume-title":"Robust Monte Carlo methods for light transport simulation","author":"Veach Eric","unstructured":"Eric Veach. 1998. Robust Monte Carlo methods for light transport simulation. Stanford University."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459804"},{"key":"e_1_2_2_53_1","volume-title":"Least-squares ray mapping method for freeform illumination optics design. Optics express","author":"Wei ShiLi","year":"2020","unstructured":"ShiLi Wei, ZhengBo Zhu, ZiChao Fan, and DingLin Ma. 2020. Least-squares ray mapping method for freeform illumination optics design. Optics express (2020)."},{"key":"e_1_2_2_54_1","volume-title":"Neural Fields in Visual Computing and Beyond. arXiv:2111.11426","author":"Xie Yiheng","year":"2021","unstructured":"Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. 2021. Neural Fields in Visual Computing and Beyond. arXiv:2111.11426 (2021)."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2580946"},{"key":"e_1_2_2_56_1","volume-title":"Path-Space Differentiable Rendering. ACM TOG","author":"Zhang Cheng","year":"2020","unstructured":"Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM TOG (2020)."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530077","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3528223.3530077","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3528223.3530077","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:25Z","timestamp":1750186945000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3528223.3530077"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,7]]}},"alternative-id":["10.1145\/3528223.3530077"],"URL":"https:\/\/doi.org\/10.1145\/3528223.3530077","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7]]},"assertion":[{"value":"2022-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}