{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T02:13:31Z","timestamp":1771467211973,"version":"3.50.1"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,7,19]]},"abstract":"<jats:p>While conventional cameras offer versatility for applications ranging from amateur photography to autonomous driving, computational cameras allow for domain-specific adaption. Cameras with co-designed optics and image processing algorithms enable high-dynamic-range image recovery, depth estimation, and hyperspectral imaging through optically encoding scene information that is otherwise undetected by conventional cameras. However, this optical encoding creates a challenging inverse reconstruction problem for conventional image recovery, and often lowers the overall photographic quality. Thus computational cameras with domain-specific optics have only been adopted in a few specialized applications where the captured information cannot be acquired in other ways. In this work, we investigate a method that combines two optical systems into one to tackle this challenge. We split the aperture of a conventional camera into two halves: one which applies an application-specific modulation to the incident light via a diffractive optical element to produce a coded image capture, and one which applies no modulation to produce a conventional image capture. Co-designing the phase modulation of the split aperture with a dual-pixel sensor allows us to simultaneously capture these coded and uncoded images without increasing physical or computational footprint. With an uncoded conventional image alongside the optically coded image in hand, we investigate image reconstruction methods that are conditioned on the conventional image, making it possible to eliminate artifacts and compute costs that existing methods struggle with. We assess the proposed method with 2-in-1 cameras for optical high-dynamic-range reconstruction, monocular depth estimation, and hyperspectral imaging, comparing favorably to all tested methods in all applications.<\/jats:p>","DOI":"10.1145\/3658225","type":"journal-article","created":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T14:47:57Z","timestamp":1721400477000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Split-Aperture 2-in-1 Computational Cameras"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9919-8816","authenticated-orcid":false,"given":"Zheng","family":"Shi","sequence":"first","affiliation":[{"name":"Princeton University, Princeton, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9136-3303","authenticated-orcid":false,"given":"Ilya","family":"Chugunov","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2676-9833","authenticated-orcid":false,"given":"Mario","family":"Bijelic","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6796-6668","authenticated-orcid":false,"given":"Geoffroi","family":"C\u00f4t\u00e9","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0186-2539","authenticated-orcid":false,"given":"Jiwoon","family":"Yeom","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6395-8521","authenticated-orcid":false,"given":"Qiang","family":"Fu","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9343-729X","authenticated-orcid":false,"given":"Hadi","family":"Amata","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4227-8508","authenticated-orcid":false,"given":"Wolfgang","family":"Heidrich","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8054-9823","authenticated-orcid":false,"given":"Felix","family":"Heide","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, United States of America"}]}],"member":"320","published-online":{"date-parts":[[2024,7,19]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58607-2_7"},{"key":"e_1_2_2_2_1","volume-title":"NTIRE 2021 challenge for defocus deblurring using dual-pixel images: Methods and results. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 578--587","author":"Abuolaim Abdullah","year":"2021","unstructured":"Abdullah Abuolaim, Radu Timofte, and Michael S Brown. 2021. NTIRE 2021 challenge for defocus deblurring using dual-pixel images: Methods and results. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 578--587."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:VISI.0000016144.56397.1a"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46478-7_2"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00231"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00265"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130896"},{"key":"e_1_2_2_8_1","volume-title":"Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288","author":"Bhat Shariq Farooq","year":"2023","unstructured":"Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M\u00fcller. 2023. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 (2023)."},{"key":"e_1_2_2_9_1","volume-title":"XII: Characterization and representation","author":"Briottet X","year":"2006","unstructured":"X Briottet, Y Boucher, A Dimmeler, A Malaplate, A Cini, Marco Diani, HHPT Bekman, P Schwering, T Skauli, I Kasen, et al. 2006. Military applications of hyperspectral imagery. In Targets and backgrounds XII: Characterization and representation, Vol. 6239. SPIE, 82--89."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00138-006-0026-2"},{"key":"e_1_2_2_11_1","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 0--0.","author":"Carvalho Marcela","year":"2018","unstructured":"Marcela Carvalho, Bertrand Le Saux, Pauline Trouv\u00e9-Peloux, Andr\u00e9s Almansa, and Fr\u00e9d\u00e9ric Champagnat. 2018. Deep Depth from Defocus: how can defocus blur improve 3D estimation using dense neural networks?. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 0--0."},{"key":"e_1_2_2_12_1","volume-title":"Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 193--200","author":"Chakrabarti A.","unstructured":"A. Chakrabarti and T. Zickler. 2011. Statistics of Real-World Hyperspectral Images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 193--200."},{"key":"e_1_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Julie Chang and Gordon Wetzstein. 2019. Deep optics for monocular depth estimation and 3d object detection. (2019) 10193--10202.","DOI":"10.1109\/ICCV.2019.01029"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01194"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401132.1401174"},{"key":"e_1_2_2_16_1","first-page":"1048","article-title":"Deep wiener deconvolution: Wiener meets deep learning for image deblurring","volume":"33","author":"Dong Jiangxin","year":"2020","unstructured":"Jiangxin Dong, Stefan Roth, and Bernt Schiele. 2020. Deep wiener deconvolution: Wiener meets deep learning for image deblurring. Advances in Neural Information Processing Systems 33 (2020), 1048--1059.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_17_1","volume-title":"A review of snapshot multidimensional optical imaging: measuring photon tags in parallel. Physics reports 616","author":"Gao Liang","year":"2016","unstructured":"Liang Gao and Lihong V Wang. 2016. A review of snapshot multidimensional optical imaging: measuring photon tags in parallel. Physics reports 616 (2016), 1--37."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00772"},{"key":"e_1_2_2_19_1","unstructured":"Carl Friedrich Gauss. 1843. Dioptric examinations by CF Gauss. in the Dieterich bookstore."},{"key":"e_1_2_2_20_1","article-title":"PS2F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing","author":"Ghanekar Bhargav","year":"2022","unstructured":"Bhargav Ghanekar, Vishwanath Saragadam, Dushyant Mehran, Anna-Karin Gustavsson, Aswin C. Sankaranarayanan, and Ashok Veeraraghavan. 2022. PS2F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI) \/ Special Issue of ICCP (August 2022).","journal-title":"IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI) \/ Special Issue of ICCP"},{"key":"e_1_2_2_21_1","doi-asserted-by":"crossref","unstructured":"Paul Green Wenyang Sun Wojciech Matusik and Fredo Durand. 2007. Multi-aperture photography. In Acm Siggraph 2007 Papers. 68--es.","DOI":"10.1145\/1275808.1276462"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00787"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1117\/1.OE.51.11.111702"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2018.2849326"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980254"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCP51581.2021.9466261"},{"key":"e_1_2_2_28_1","doi-asserted-by":"crossref","unstructured":"Daniel S Jeon Seung-Hwan Baek Shinyoung Yi Qiang Fu Xiong Dun Wolfgang Heidrich and Min H Kim. 2019. Compact snapshot hyperspectral imaging with diffracted rotation. (2019).","DOI":"10.1145\/3306346.3322946"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/GlobalSIP45357.2019.8969167"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.3169\/mta.4.123"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.3844\/jcssp.2007.419.423"},{"key":"e_1_2_2_32_1","volume-title":"Image and depth from a conventional camera with a coded aperture. ACM transactions on graphics (TOG) 26, 3","author":"Levin Anat","year":"2007","unstructured":"Anat Levin, Rob Fergus, Fr\u00e9do Durand, and William T Freeman. 2007. Image and depth from a conventional camera with a coded aperture. ACM transactions on graphics (TOG) 26, 3 (2007), 70--es."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01916"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1364\/PRJ.450799"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00172"},{"key":"e_1_2_2_36_1","unstructured":"Kris Malkiewicz and M David Mullen. 2009. Cinematography. Simon and Schuster."},{"key":"e_1_2_2_37_1","volume-title":"IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). http:\/\/lmb.informatik.uni-freiburg.de\/Publications\/2016\/MIFDB16 arXiv:1512","author":"Mayer N.","unstructured":"N. Mayer, E. Ilg, P. H\u00e4usser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. 2016. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). http:\/\/lmb.informatik.uni-freiburg.de\/Publications\/2016\/MIFDB16 arXiv:1512.02134."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00145"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.3390\/rs71115467"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.258"},{"key":"e_1_2_2_41_1","volume-title":"Deep-STORM3D: dense 3D localization microscopy and PSF design by deep learning. Nature methods 17, 7","author":"Nehme Elias","year":"2020","unstructured":"Elias Nehme, Daniel Freedman, Racheli Gordon, Boris Ferdman, Lucien E Weiss, Onit Alalouf, Tal Naor, Reut Orange, Tomer Michaeli, and Yoav Shechtman. 2020. Deep-STORM3D: dense 3D localization microscopy and PSF design by deep learning. Nature methods 17, 7 (2020), 734--740."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00432"},{"key":"e_1_2_2_43_1","volume-title":"Three-Dimensional and Multidimensional Microscopy: Image Acquisition and Processing XVI","author":"Prasanna Pavani Sri Rama","unstructured":"Sri Rama Prasanna Pavani and Rafael Piestun. 2009. 3D microscopy with a double-helix point spread function. In Three-Dimensional and Multidimensional Microscopy: Image Acquisition and Processing XVI, Vol. 7184. SPIE, 65--71."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356526"},{"key":"e_1_2_2_45_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Punnappurath Abhijith","unstructured":"Abhijith Punnappurath and Michael S. Brown. 2019. Reflection Removal Using a Dual-Pixel Sensor. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01196"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3019967"},{"key":"e_1_2_2_48_1","unstructured":"Erik Reinhard Wolfgang Heidrich Paul Debevec Sumanta Pattanaik Greg Ward and Karol Myszkowski. 2010. High dynamic range imaging: acquisition display and image-based lighting. Morgan Kaufmann."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995335"},{"key":"e_1_2_2_50_1","volume-title":"Tsang Ing Ren, and Nima Khademi Kalantari","author":"Santos Marcel Santana","year":"2020","unstructured":"Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. arXiv preprint arXiv:2005.07335 (2020)."},{"key":"e_1_2_2_51_1","volume-title":"Optimal point spread function design for 3D imaging. Physical review letters 113, 13","author":"Shechtman Yoav","year":"2014","unstructured":"Yoav Shechtman, Steffen J Sahl, Adam S Backer, and William E Moerner. 2014. Optimal point spread function design for 3D imaging. Physical review letters 113, 13 (2014), 133902."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530185"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201333"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00146"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3446791"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2022.3212837"},{"key":"e_1_2_2_57_1","volume-title":"IEEE 6th Workshop on Omnidirectional Vision, Camera Networks and Non-classical Cameras","author":"Wang Hongcheng","year":"2005","unstructured":"Hongcheng Wang, Ramesh Raskar, and Narendra Ahuja. 2005. High dynamic range video using split aperture camera. In IEEE 6th Workshop on Omnidirectional Vision, Camera Networks and Non-classical Cameras, Washington, DC, USA. Citeseer."},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPHOT.2019.8747330"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00223"},{"key":"e_1_2_2_60_1","unstructured":"F. Yasuma T. Mitsunaga D. Iso and S.K. Nayar. 2008. Generalized Assorted Pixel Camera: Post-Capture Control of Resolution Dynamic Range and Spectrum. Technical Report."},{"key":"e_1_2_2_61_1","volume-title":"Massimiliano Di Pietro, Wladyslaw Januszewicz, Rebecca C Fitzgerald, and Sarah E Bohndiek.","author":"Yoon Jonghee","year":"2019","unstructured":"Jonghee Yoon, James Joseph, Dale J Waterhouse, A Siri Luthman, George SD Gordon, Massimiliano Di Pietro, Wladyslaw Januszewicz, Rebecca C Fitzgerald, and Sarah E Bohndiek. 2019. A clinically translatable hyperspectral endoscopy (HySE) system for imaging the gastrointestinal tract. Nature communications 10, 1 (2019), 1--13."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00219"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658225","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3658225","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:16Z","timestamp":1750291456000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658225"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,19]]},"references-count":62,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7,19]]}},"alternative-id":["10.1145\/3658225"],"URL":"https:\/\/doi.org\/10.1145\/3658225","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,19]]},"assertion":[{"value":"2024-07-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}