{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T17:13:22Z","timestamp":1782321202738,"version":"3.54.5"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,4,30]],"date-time":"2021-04-30T00:00:00Z","timestamp":1619740800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2021,4,30]]},"abstract":"<jats:p>\n            Most modern commodity imaging systems we use directly for photography\u2014or indirectly rely on for downstream applications\u2014employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today\u2019s compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss\u2019 linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize\n            <jats:italic>entire<\/jats:italic>\n            lens systems\n            <jats:italic>jointly<\/jats:italic>\n            with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-to-end losses. To this end, we propose a learned,\n            <jats:italic>differentiable forward model for compound optics<\/jats:italic>\n            and an\n            <jats:italic>alternating proximal optimization<\/jats:italic>\n            method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as\n            <jats:sc>Zemax<\/jats:sc>\n            . We can thus assess our method across many camera system designs and end-to-end applications. We validate our approach in an automotive camera optics setting\u2014together with hardware ISP post processing and detection\u2014outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-tuning methods qualitatively and quantitatively, across\n            <jats:italic>all<\/jats:italic>\n            domain-specific applications tested.\n          <\/jats:p>","DOI":"10.1145\/3446791","type":"journal-article","created":{"date-parts":[[2021,6,21]],"date-time":"2021-06-21T20:08:32Z","timestamp":1624306112000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":120,"title":["Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design"],"prefix":"10.1145","volume":"40","author":[{"given":"Ethan","family":"Tseng","sequence":"first","affiliation":[{"name":"Princeton University, Princeton, NJ"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Mosleh","sequence":"additional","affiliation":[{"name":"Algolux, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fahim","family":"Mannan","sequence":"additional","affiliation":[{"name":"Algolux, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karl","family":"St-Arnaud","sequence":"additional","affiliation":[{"name":"Algolux, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avinash","family":"Sharma","sequence":"additional","affiliation":[{"name":"Algolux, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifan","family":"Peng","sequence":"additional","affiliation":[{"name":"Algolux, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Braun","sequence":"additional","affiliation":[{"name":"Hosch\u00fcle Dusseldorf"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Derek","family":"Nowrouzezahrai","sequence":"additional","affiliation":[{"name":"McGill University, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Fran\u00e7ois","family":"Lalonde","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Laval, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Heide","sequence":"additional","affiliation":[{"name":"Princeton University and Algolux, QC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,6,21]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.905752"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995413"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-30619-y"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.01029"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00347"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.273"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCE.2014.6884437"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/2319074.2321415"},{"key":"e_1_2_2_9_1","first-page":"1","article-title":"Neural architecture search: A survey","volume":"20","author":"Elsken Thomas","year":"2019","unstructured":"Thomas Elsken , Jan Hendrik Metzen , and Frank Hutter . 2019 . Neural architecture search: A survey . Journal of Machine Learning Research 20 , 55 (2019), 1 \u2013 21 . Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. Journal of Machine Learning Research 20, 55 (2019), 1\u201321.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275081"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cirp.2013.05.003"},{"key":"e_1_2_2_12_1","volume-title":"Introduction to Modern Optics","author":"Fowles Grant R.","unstructured":"Grant R. Fowles . 1989. Introduction to Modern Optics . Courier Corporation . Grant R. Fowles. 1989. Introduction to Modern Optics. Courier Corporation."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8460737"},{"key":"e_1_2_2_14_1","volume-title":"Photopic Eye Response Filter. Retrieved","year":"2021","unstructured":"Galvoptics. 2020. Photopic Eye Response Filter. Retrieved April 2, 2021 from https:\/\/www.galvoptics.co.uk\/optical-components\/optical-filters\/photopic-eye-response-filter\/ Galvoptics. 2020. Photopic Eye Response Filter. Retrieved April 2, 2021 from https:\/\/www.galvoptics.co.uk\/optical-components\/optical-filters\/photopic-eye-response-filter\/"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.617680"},{"key":"e_1_2_2_16_1","unstructured":"Carl Friedrich Gauss. 1841. Dioptrische Untersuchungen. Dieterich.  Carl Friedrich Gauss. 1841. Dioptrische Untersuchungen. Dieterich."},{"key":"e_1_2_2_17_1","unstructured":"Joseph M. Geary. 2002. Introduction to Lens Design: With Practical ZEMAX Examples. Willmann-Bell Richmond.  Joseph M. Geary. 2002. Introduction to Lens Design: With Practical ZEMAX Examples. Willmann-Bell Richmond."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913491297"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982399"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073592"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/280814.280816"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12301"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1117\/1.OE.54.3.035105"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980254"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601174"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661260"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2010.5540158"},{"key":"e_1_2_2_28_1","volume-title":"Photography - Electronic still picture imaging \u2013 Resolution and spatial frequency responses. 1\u201349. Retrieved","author":"ISO.","year":"2019","unstructured":"ISO. 2017. ISO 71696. Photography - Electronic still picture imaging \u2013 Resolution and spatial frequency responses. 1\u201349. Retrieved January 5, 2019 from https:\/\/www.iso.org\/standard\/71696.html ISO. 2017. ISO 71696. Photography - Electronic still picture imaging \u2013 Resolution and spatial frequency responses. 1\u201349. Retrieved January 5, 2019 from https:\/\/www.iso.org\/standard\/71696.html"},{"key":"e_1_2_2_29_1","volume-title":"Fundamental Optical Design","author":"Kidger Michael J.","unstructured":"Michael J. Kidger . 2002. Fundamental Optical Design . SPIE Press . Michael J. Kidger. 2002. Fundamental Optical Design. SPIE Press."},{"key":"e_1_2_2_30_1","volume-title":"Johnson","author":"Kingslake Rudolf","year":"2009","unstructured":"Rudolf Kingslake and Roger B . Johnson . 2009 . Lens Design Fundamentals. Academic Press . Rudolf Kingslake and Roger B. Johnson. 2009. Lens Design Fundamentals. Academic Press."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/218380.218463"},{"key":"e_1_2_2_32_1","unstructured":"Kowa. 2020. LM6NCL. Retrieved April 2 2021 from https:\/\/lenses.kowa-usa.com\/ncl-series\/490-lm6ncl.html  Kowa. 2020. LM6NCL. Retrieved April 2 2021 from https:\/\/lenses.kowa-usa.com\/ncl-series\/490-lm6ncl.html"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201383"},{"key":"e_1_2_2_34_1","volume-title":"Handbook of Optical Design","author":"Malacara-Hern\u00e1ndez Daniel","unstructured":"Daniel Malacara-Hern\u00e1ndez and Zacar\u00edas Malacara-Hern\u00e1ndez . 2016. Handbook of Optical Design . CRC Press , Boca Raton, FL . Daniel Malacara-Hern\u00e1ndez and Zacar\u00edas Malacara-Hern\u00e1ndez. 2016. Handbook of Optical Design. CRC Press, Boca Raton, FL."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00145"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00755"},{"key":"e_1_2_2_37_1","volume-title":"1\/2.5-Inch 5 Mp CMOS Digital Image Sensor. Retreived on","author":"Semiconductor ON","year":"2021","unstructured":"ON Semiconductor . 2015. MT9P001 : 1\/2.5-Inch 5 Mp CMOS Digital Image Sensor. Retreived on 03 May , 2021 from https:\/\/www.onsemi.com\/pdf\/datasheet\/mt9p001-d.pdf. ON Semiconductor. 2015. MT9P001: 1\/2.5-Inch 5 Mp CMOS Digital Image Sensor. Retreived on 03 May, 2021 from https:\/\/www.onsemi.com\/pdf\/datasheet\/mt9p001-d.pdf."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2018.8451818"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356526"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/3208704"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2005.1407713"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969239.2969250"},{"key":"e_1_2_2_43_1","first-page":"322","article-title":"Potential and challenges of using computer graphics for the simulation of optical measurement systems.GMA\/ITG Fachtagung","volume":"18","author":"Retzlaff Max-Gerd","year":"2016","unstructured":"Max-Gerd Retzlaff , Johannes Hanika , J\u00fcrgen Beyerer , and Carsten Dachsbacher . 2016 . Potential and challenges of using computer graphics for the simulation of optical measurement systems.GMA\/ITG Fachtagung : Sensoren und Messsysteme 18 (2016), 322 \u2013 329 . Max-Gerd Retzlaff, Johannes Hanika, J\u00fcrgen Beyerer, and Carsten Dachsbacher. 2016. Potential and challenges of using computer graphics for the simulation of optical measurement systems.GMA\/ITG Fachtagung: Sensoren und Messsysteme 18 (2016), 322\u2013329.","journal-title":"Sensoren und Messsysteme"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/3071773.3071783"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2013.2278548"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33765-9_4"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201333"},{"key":"e_1_2_2_48_1","volume-title":"Abberation and Optical Design Theory","author":"Sliusarev Georgii Georgievich","unstructured":"Georgii Georgievich Sliusarev . 1984. Abberation and Optical Design Theory ( 2 nd ed.). Adam Higler Ltd ., Bristol, England. Georgii Georgievich Sliusarev. 1984. Abberation and Optical Design Theory (2nd ed.). Adam Higler Ltd., Bristol, England.","edition":"2"},{"key":"e_1_2_2_49_1","volume-title":"Modern Lens Design","author":"Smith Warren J.","unstructured":"Warren J. Smith . 2005. Modern Lens Design ( 2 nd ed.). McGraw-Hill , New York, NY . Warren J. Smith. 2005. Modern Lens Design (2nd ed.). McGraw-Hill, New York, NY.","edition":"2"},{"key":"e_1_2_2_50_1","unstructured":"EMVA. 2010. Standard for Characterization of image Sensors and Cameras. Release 3.https:\/\/www.emva.org\/wp-content\/uploads\/EMVA1288-3.1a.pdf.  EMVA. 2010. Standard for Characterization of image Sensors and Cameras. Release 3.https:\/\/www.emva.org\/wp-content\/uploads\/EMVA1288-3.1a.pdf."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2011.01851.x"},{"key":"e_1_2_2_52_1","first-page":"4","article-title":"Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors","volume":"7","author":"Stork David G.","year":"2014","unstructured":"David G. Stork and Patrick R. Gill . 2014 . Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors . International Journal on Advances in Systems and Measurements 7 , 3 (2014), 4 . David G. Stork and Patrick R. Gill. 2014. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements 7, 3 (2014), 4.","journal-title":"International Journal on Advances in Systems and Measurements"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1201\/9781315181448"},{"key":"e_1_2_2_54_1","unstructured":"Libin Sun Neel Joshi Brian Guenter and James Hays. 2015. Lens factory: Automatic lens generation using off-the-shelf components. arXiv:1506.08956  Libin Sun Neel Joshi Brian Guenter and James Hays. 2015. Lens factory: Automatic lens generation using off-the-shelf components. arXiv:1506.08956"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00146"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/938978.939190"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322996"},{"key":"e_1_2_2_58_1","volume-title":"Optical Engineering Fundamentals","author":"Walker Bruce H.","unstructured":"Bruce H. Walker . 2008. Optical Engineering Fundamentals . Vol. 82 . SPIE Press , Bellingham, WA . Bruce H. Walker. 2008. Optical Engineering Fundamentals. Vol. 82. SPIE Press, Bellingham, WA."},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045296"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1137\/120887795"},{"key":"e_1_2_2_61_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.","author":"Yu Fisher","year":"2020","unstructured":"Fisher Yu , Haofeng Chen , Xin Wang , Wenqi Xian , Yingying Chen , Fangchen Liu , Vashisht Madhavan , and Trevor Darrell . 2020 . BDD100K: A diverse driving dataset for heterogeneous multitask learning . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12204-016-1709-0"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1117\/1.3600632"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00068"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3446791","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3446791","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:31Z","timestamp":1750193251000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3446791"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,30]]},"references-count":64,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,4,30]]}},"alternative-id":["10.1145\/3446791"],"URL":"https:\/\/doi.org\/10.1145\/3446791","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,30]]},"assertion":[{"value":"2020-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}