{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,29]],"date-time":"2026-03-29T01:37:42Z","timestamp":1774748262980,"version":"3.50.1"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,7,30]],"date-time":"2018-07-30T00:00:00Z","timestamp":1532908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2018,8,31]]},"abstract":"<jats:p>In typical cameras the optical system is designed first; once it is fixed, the parameters in the image processing algorithm are tuned to get good image reproduction. In contrast to this sequential design approach, we consider joint optimization of an optical system (for example, the physical shape of the lens) together with the parameters of the reconstruction algorithm. We build a fully-differentiable simulation model that maps the true source image to the reconstructed one. The model includes diffractive light propagation, depth and wavelength-dependent effects, noise and nonlinearities, and the image post-processing. We jointly optimize the optical parameters and the image processing algorithm parameters so as to minimize the deviation between the true and reconstructed image, over a large set of images. We implement our joint optimization method using autodifferentiation to efficiently compute parameter gradients in a stochastic optimization algorithm. We demonstrate the efficacy of this approach by applying it to achromatic extended depth of field and snapshot super-resolution imaging.<\/jats:p>","DOI":"10.1145\/3197517.3201333","type":"journal-article","created":{"date-parts":[[2018,7,31]],"date-time":"2018-07-31T15:56:23Z","timestamp":1533052583000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":376,"title":["End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging"],"prefix":"10.1145","volume":"37","author":[{"given":"Vincent","family":"Sitzmann","sequence":"first","affiliation":[{"name":"Stanford University"}]},{"given":"Steven","family":"Diamond","sequence":"additional","affiliation":[{"name":"Stanford University"}]},{"given":"Yifan","family":"Peng","sequence":"additional","affiliation":[{"name":"The University of British Columbia and Canada and Stanford University"}]},{"given":"Xiong","family":"Dun","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology, Saudi Arabia"}]},{"given":"Stephen","family":"Boyd","sequence":"additional","affiliation":[{"name":"Stanford University"}]},{"given":"Wolfgang","family":"Heidrich","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology, Saudi Arabia"}]},{"given":"Felix","family":"Heide","sequence":"additional","affiliation":[{"name":"Stanford University"}]},{"given":"Gordon","family":"Wetzstein","sequence":"additional","affiliation":[{"name":"Stanford University"}]}],"member":"320","published-online":{"date-parts":[[2018,7,30]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Proc. IEEE ICCP. 1--11","author":"Antipa N."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2016.2593662"},{"key":"e_1_2_2_3_1","volume-title":"Proc. CVPR. 135--142","author":"Ben-Ezra M."},{"key":"e_1_2_2_4_1","volume-title":"Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light (7 ed.)","author":"Born Max"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature11150"},{"key":"e_1_2_2_6_1","unstructured":"Ayan Chakrabarti. 2016. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems. 3081--3089.   Ayan Chakrabarti. 2016. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems. 3081--3089."},{"key":"e_1_2_2_7_1","volume-title":"A broadband achromatic metalens for focusing and imaging in the visible. Nature nanotechnology","author":"Chen Wei Ting","year":"2018"},{"key":"e_1_2_2_8_1","volume-title":"Metasurface optics for full-color computational imaging. Science Advances 4, 2","author":"Colburn Shane","year":"2018"},{"key":"e_1_2_2_9_1","volume-title":"Proc. ICCP.","author":"Cossairt O."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPHOT.2010.5585101"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778768"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1364\/OE.23.010224"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/258734.258884"},{"key":"e_1_2_2_14_1","volume-title":"Dirty pixels: Optimizing image classification architectures for raw sensor data. arXiv preprint arXiv:1701.06487","author":"Diamond Steven","year":"2017"},{"key":"e_1_2_2_15_1","unstructured":"Steven Diamond Vincent Sitzmann Felix Heide and Gordon Wetzstein. 2017b. Unrolled Optimization with Deep Priors. (2017). arXiv:1705.08041  Steven Diamond Vincent Sitzmann Felix Heide and Gordon Wetzstein. 2017b. Unrolled Optimization with Deep Priors. (2017). arXiv:1705.08041"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2439281"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46475-6_25"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1364\/AO.34.001859"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130816"},{"key":"e_1_2_2_20_1","volume-title":"Achromatic hybrid refractive-diffractive lens with extended depth of focus. Applied optics 43, 30","author":"Flores Angel","year":"2004"},{"key":"e_1_2_2_21_1","volume-title":"Introduction to Fourier Optics (4 ed.)","author":"Goodman Joseph"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980254"},{"key":"e_1_2_2_23_1","volume-title":"Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6, 33543","author":"Heide Felix","year":"2016"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299156"},{"key":"e_1_2_2_25_1","volume-title":"Deepbinarymask: Learning a binary mask for video compressive sensing. arXiv preprint arXiv:1607.03343","author":"Iliadis Michael","year":"2016"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88682-2_24"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073609"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.182"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.181"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.618"},{"key":"e_1_2_2_31_1","volume-title":"Land and Dan-Eric Nielsson","author":"Michael","year":"2002"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276464"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531326.1531403"},{"key":"e_1_2_2_34_1","unstructured":"Zhiqiang Liu. 2007. Diffractive lens with extended depth of focus and its applications. (2007).  Zhiqiang Liu. 2007. Diffractive lens with extended depth of focus and its applications. (2007)."},{"key":"e_1_2_2_35_1","volume-title":"Proceedings of IS&T 442--448","author":"Mann S."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2001.937655"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461914"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/JMEMS.2003.823220"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.258"},{"key":"e_1_2_2_40_1","first-page":"372","article-title":"A method of solving a convex programming problem with convergence rate O(1\/k<sup>2<\/sup>)","volume":"27","author":"Nesterov Y.","year":"1983","journal-title":"Soviet Mathematics Doklady"},{"key":"e_1_2_2_42_1","doi-asserted-by":"crossref","unstructured":"Sri Rama Prasanna Pavani Michael A. Thompson Julie S. Biteen Samuel J. Lord Na Liu Robert J. Twieg Rafael Piestun and W. E. Moerner. 2009. Three-dimensional single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. 106 9 (2009) 2995--2999.  Sri Rama Prasanna Pavani Michael A. Thompson Julie S. Biteen Samuel J. Lord Na Liu Robert J. Twieg Rafael Piestun and W. E. Moerner. 2009. Three-dimensional single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. 106 9 (2009) 2995--2999.","DOI":"10.1073\/pnas.0900245106"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925941"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141911.1141957"},{"key":"e_1_2_2_45_1","doi-asserted-by":"crossref","unstructured":"Erik Reinhard Greg Ward Sumanta Pattanaik and Paul Debevec. 2005. High Dynamic Range Imaging: Acquisition Display and Image-Based Lighting. Morgan Kaufmann.   Erik Reinhard Greg Ward Sumanta Pattanaik and Paul Debevec. 2005. High Dynamic Range Imaging: Acquisition Display and Image-Based Lighting. Morgan Kaufmann.","DOI":"10.1016\/B978-012585263-0\/50010-5"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995335"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299003"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601200"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevLett.113.133902"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.207"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33765-9_4"},{"key":"e_1_2_2_52_1","volume-title":"Modern Optical Engineering (4 ed.)","author":"Smith Warren J."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.246"},{"key":"e_1_2_2_54_1","volume-title":"Lens Factory: Automatic Lens Generation Using Off-the-shelf Components. CoRR abs\/1506.08956","author":"Sun Libin","year":"2015"},{"key":"e_1_2_2_55_1","volume-title":"Proceedings of Asian Conference on Computer Vision. 111--126","author":"Timofte Radu","year":"2014"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276463"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1364\/AO.47.000B44"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073614"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.5555\/2919332.2919871"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2011.02073.x"},{"key":"e_1_2_2_61_1","volume-title":"Freeform illumination design: a nonlinear boundary problem for the elliptic Monge-Amp\u00e9re equation. Optics letters 38, 2","author":"Wu Rengmao","year":"2013"},{"key":"e_1_2_2_62_1","volume-title":"Ce Liu, and Jiaya Jia.","author":"Xu Li","year":"2014"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1063\/1.4983831"},{"key":"e_1_2_2_64_1","volume-title":"ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701","author":"Zeiler Matthew D","year":"2012"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.484"},{"key":"e_1_2_2_66_1","volume-title":"Proc. ICCV.","author":"Zhou C."},{"key":"e_1_2_2_67_1","volume-title":"Proc. IEEE ICCP. 1--8.","author":"Zhou C."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3197517.3201333","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3197517.3201333","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:39:44Z","timestamp":1750210784000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3197517.3201333"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,30]]},"references-count":66,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,8,31]]}},"alternative-id":["10.1145\/3197517.3201333"],"URL":"https:\/\/doi.org\/10.1145\/3197517.3201333","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,7,30]]},"assertion":[{"value":"2018-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}