{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T23:08:03Z","timestamp":1783552083485,"version":"3.55.0"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T00:00:00Z","timestamp":1690329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CAREER Award 1943149"],"award-info":[{"award-number":["CAREER Award 1943149"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation","doi-asserted-by":"crossref","award":["166289"],"award-info":[{"award-number":["166289"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>Megapixel single-photon avalanche diode (SPAD) arrays have been developed recently, opening up the possibility of deploying SPADs as generalpurpose passive cameras for photography and computer vision. However, most previous work on SPADs has been limited to monochrome imaging. We propose a computational photography technique that reconstructs high-quality color images from mosaicked binary frames captured by a SPAD array, even for high-dyanamic-range (HDR) scenes with complex and rapid motion. Inspired by conventional burst photography approaches, we design algorithms that jointly denoise and demosaick single-photon image sequences. Based on the observation that motion effectively increases the color sample rate, we design a blue-noise pseudorandom RGBW color filter array for SPADs, which is tailored for imaging dark, dynamic scenes. Results on simulated data, as well as real data captured with a fabricated color SPAD hardware prototype shows that the proposed method can reconstruct high-quality images with minimal color artifacts even for challenging low-light, HDR and fast-moving scenes. We hope that this paper, by adding color to computational single-photon imaging, spurs rapid adoption of SPADs for real-world passive imaging applications.<\/jats:p>","DOI":"10.1145\/3592438","type":"journal-article","created":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T14:29:21Z","timestamp":1690381761000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Seeing Photons in Color"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0092-9744","authenticated-orcid":false,"given":"Sizhuo","family":"Ma","sequence":"first","affiliation":[{"name":"University of Wisconsin-Madison, Madison, United States of America"},{"name":"Snap Inc., New York, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8261-3630","authenticated-orcid":false,"given":"Varun","family":"Sundar","sequence":"additional","affiliation":[{"name":"University of Wisconsin-Madison, Madison, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5909-5147","authenticated-orcid":false,"given":"Paul","family":"Mos","sequence":"additional","affiliation":[{"name":"EPFL, Neuch\u00e2tel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6636-6596","authenticated-orcid":false,"given":"Claudio","family":"Bruschini","sequence":"additional","affiliation":[{"name":"EPFL, Neuch\u00e2tel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0620-3365","authenticated-orcid":false,"given":"Edoardo","family":"Charbon","sequence":"additional","affiliation":[{"name":"EPFL, Neuch\u00e2tel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2323-7700","authenticated-orcid":false,"given":"Mohit","family":"Gupta","sequence":"additional","affiliation":[{"name":"University of Wisconsin-Madison, Madison, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,7,26]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/40.743681"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2004.841200"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TED.2015.2458295"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.image.2019.05.018"},{"key":"e_1_2_2_5_1","first-page":"971","article-title":"Color imaging array","volume":"3","author":"Bayer Bryce E","year":"1976","unstructured":"Bryce E Bayer. 1976. Color imaging array. US Patent 3,971,065.","journal-title":"US Patent"},{"key":"e_1_2_2_6_1","volume-title":"Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. In IEEE\/CVF International Conference on Computer Vision (ICCV). 2460--2470","author":"Bhat Goutam","year":"2021","unstructured":"Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. 2021. Deep Reparametrization of Multi-Frame Super-Resolution and Denoising. In IEEE\/CVF International Conference on Computer Vision (ICCV). 2460--2470. arXiv:2108.08286 [cs, eess]"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10851-007-0017-6"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1364\/opticaopen.22090487.v2"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41377-019-0191-5"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1364\/OE.23.020997"},{"key":"e_1_2_2_11_1","volume-title":"International Conference on Neural Information Processing Systems (NIPS'16)","author":"Chakrabarti Ayan","year":"2016","unstructured":"Ayan Chakrabarti. 2016. Learning Sensor Multiplexing Design through Back-propagation. In International Conference on Neural Information Processing Systems (NIPS'16). Curran Associates Inc., Barcelona, Spain, 3089--3097."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPHOT.2014.6831801"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.3390\/s16111961"},{"key":"e_1_2_2_14_1","volume-title":"IEEE International Conference on Computational Photography (ICCP)","author":"Chandramouli Paramanand","year":"2019","unstructured":"Paramanand Chandramouli, Samuel Burri, Claudio Bruschini, Edoardo Charbon, and Andreas Kolb. 2019. A Bit Too Much? High Speed Imaging from Sparse Photon Counts. In IEEE International Conference on Computational Photography (ICCP). Tokyo, Japan, 1--9. arXiv:1811.02396"},{"key":"e_1_2_2_15_1","volume-title":"Seeing Motion in the Dark. In International Conference on Computer Vision (ICCV). 3185--3194","author":"Chen Chen","year":"2019","unstructured":"Chen Chen, Qifeng Chen, Minh N Do, and Vladlen Koltun. 2019. Seeing Motion in the Dark. In International Conference on Computer Vision (ICCV). 3185--3194."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00347"},{"key":"e_1_2_2_17_1","volume-title":"Chan","author":"Chi Yiheng","year":"2020","unstructured":"Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, and Stanley H. Chan. 2020. Dynamic Low-Light Imaging with Quanta Image Sensors. In European Conference on Computer Vision (ECCV), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 122--138."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8461685"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2009.5413388"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2009.12.004"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2007.901238"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/325165.325182"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01691"},{"key":"e_1_2_2_24_1","volume-title":"Burst Image Restoration and Enhancement. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5759--5768","author":"Dudhane Akshay","year":"2022","unstructured":"Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Burst Image Restoration and Enhancement. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5759--5768. arXiv:2110.03680 [cs]"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2021.3052694"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2020.2964238"},{"key":"e_1_2_2_27_1","volume-title":"IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors. 214--217","author":"Fossum Eric R.","year":"2005","unstructured":"Eric R. Fossum. 2005. What To Do With Sub-Diffraction Limit (SDL) Pixels?---A Proposal for a Gigapixel Digital Film Sensor (DFS). In IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors. 214--217."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/JEDS.2013.2284054"},{"key":"e_1_2_2_29_1","first-page":"476","article-title":"Sparsely sampled image sensing device with color and luminance photosites","volume":"6","author":"Gindele Edward B","year":"2002","unstructured":"Edward B Gindele and Andrew C Gallagher. 2002. Sparsely sampled image sensing device with color and luminance photosites. US Patent 6,476,865.","journal-title":"US Patent"},{"key":"e_1_2_2_30_1","volume-title":"Image Classification in the Dark Using Quanta Image Sensors. In European Conference on Computer Vision (ECCV). Springer, 484--501","author":"Gnanasambandam Abhiram","year":"2006","unstructured":"Abhiram Gnanasambandam and Stanley H. Chan. 2020. Image Classification in the Dark Using Quanta Image Sensors. In European Conference on Computer Vision (ECCV). Springer, 484--501. arXiv:2006.02026"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1364\/OE.27.017298"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01267-0_33"},{"key":"e_1_2_2_33_1","volume-title":"Robert K Henderson, Johannes Herrnsdorf, Martin D Dawson, and Michael J Strain.","author":"Griffiths Alexander D","year":"2019","unstructured":"Alexander D Griffiths, Haochang Chen, David Day-Uei Li, Robert K Henderson, Johannes Herrnsdorf, Martin D Dawson, and Michael J Strain. 2019. Multispectral time-of-flight imaging using light-emitting diodes. Optics express 27, 24 (2019), 35485--35498."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.3390\/s18020323"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980254"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925875"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661260"},{"key":"e_1_2_2_38_1","volume-title":"Real-Time Intermediate Flow Estimation for Video Frame Interpolation. In European Conference on Computer Vision (ECCV). Springer, 624--642","author":"Huang Zhewei","year":"2022","unstructured":"Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. 2022. Real-Time Intermediate Flow Estimation for Video Frame Interpolation. In European Conference on Computer Vision (ECCV). Springer, 624--642. arXiv:2011.06294 [cs]"},{"key":"e_1_2_2_39_1","volume-title":"Passive Inter-Photon Imaging. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8585--8595","author":"Ingle Atul","year":"2021","unstructured":"Atul Ingle, Trevor Seets, Mauro Buttafava, Shantanu Gupta, Alberto Tosi, Mohit Gupta, and Andreas Velten. 2021. Passive Inter-Photon Imaging. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8585--8595."},{"key":"e_1_2_2_40_1","volume-title":"High Flux Passive Imaging with Single-Photon Sensors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6760--6769","author":"Ingle Atul","year":"2019","unstructured":"Atul Ingle, Andreas Velten, and Mohit Gupta. 2019. High Flux Passive Imaging with Single-Photon Sensors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6760--6769."},{"key":"e_1_2_2_41_1","unstructured":"ITU-R. 2011. Studio Encoding Parameters of Digital Television for Standard 4:3 and Wide-Screen 16:9 Aspect Ratios. (2011)."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3059293"},{"key":"e_1_2_2_43_1","volume-title":"International Image Sensor Workshop (IISW).","author":"Iwabuchi Kiyotaka","year":"2019","unstructured":"Kiyotaka Iwabuchi, Tomohiro Yamazaki, and Takayuki Hamamoto. 2019. Iterative Image Reconstruction for Quanta Image Sensor by Using Variance-based Motion Estimation. In International Image Sensor Workshop (IISW)."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00742"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1364\/AO.391579"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2021.3125394"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6248068"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2010324.1964937"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00608"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1117\/1.3116562"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.3390\/s20123423"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2016.2633869"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58595-2_10"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356508"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661277"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1364\/OPTICA.4.001474"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-17952-z"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392470"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2012.2199324"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1166087.1166095"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00265"},{"key":"e_1_2_2_62_1","volume-title":"Barron","author":"Mildenhall Ben","year":"2021","unstructured":"Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T. Barron. 2021. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. arXiv:2111.13679 [cs, eess] (Nov. 2021). arXiv:2111.13679 [cs, eess]"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1364\/OPTICA.386574"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/IEDM19574.2021.9720605"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSSC.2021.3114620"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.3390\/s17071523"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSCC42614.2022.9731644"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2006.312669"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.806367"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01234"},{"key":"e_1_2_2_71_1","volume-title":"Wavelength-time coding for multispectral 3D imaging using single-photon LiDAR. Optics express 26, 23","author":"Ren Ximing","year":"2018","unstructured":"Ximing Ren, Yoann Altmann, Rachael Tobin, Aongus Mccarthy, Stephen Mclaughlin, and Gerald S Buller. 2018. Wavelength-time coding for multispectral 3D imaging using single-photon LiDAR. Optics express 26, 23 (2018), 30146--30161."},{"key":"e_1_2_2_72_1","volume-title":"Motion Adaptive Deblurring with Single-Photon Cameras. In IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV). 1945--1954","author":"Seets Trevor","year":"2021","unstructured":"Trevor Seets, Atul Ingle, Martin Laurenzis, and Andreas Velten. 2021. Motion Adaptive Deblurring with Single-Photon Cameras. In IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV). 1945--1954. arXiv:2012.07931"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1364\/OPTICA.389905"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1364\/OE.27.023661"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01422"},{"key":"e_1_2_2_76_1","first-page":"314","article-title":"Image processing device, image processing method, and program pertaining to image correction","volume":"8","author":"Tachi Masayuki","year":"2012","unstructured":"Masayuki Tachi. 2012. Image processing device, image processing method, and program pertaining to image correction. US Patent 8,314,863.","journal-title":"US Patent"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2006.888330"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTQE.2018.2867439"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323024"},{"key":"e_1_2_2_80_1","volume-title":"Basis Prediction Networks for Effective Burst Denoising with Large Kernels. arXiv:1912.04421 [cs] (Dec","author":"Xia Zhihao","year":"2019","unstructured":"Zhihao Xia, Federico Perazzi, Micha\u00ebl Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. 2019. Basis Prediction Networks for Effective Burst Denoising with Large Kernels. arXiv:1912.04421 [cs] (Dec. 2019). arXiv:1912.04421 [cs]"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2011.2179306"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sna.2007.06.021"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01182"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592438","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3592438","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3592438","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:59Z","timestamp":1750182539000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592438"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,26]]},"references-count":83,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.1145\/3592438"],"URL":"https:\/\/doi.org\/10.1145\/3592438","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,26]]},"assertion":[{"value":"2023-07-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}