{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T04:19:15Z","timestamp":1783570755528,"version":"3.55.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,7,30]],"date-time":"2018-07-30T00:00:00Z","timestamp":1532908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2018,8,31]]},"abstract":"<jats:p>Sensors which capture 3D scene information provide useful data for tasks in vehicle navigation, gesture recognition, human pose estimation, and geometric reconstruction. Active illumination time-of-flight sensors in particular have become widely used to estimate a 3D representation of a scene. However, the maximum range, density of acquired spatial samples, and overall acquisition time of these sensors is fundamentally limited by the minimum signal required to estimate depth reliably. In this paper, we propose a data-driven method for photon-efficient 3D imaging which leverages sensor fusion and computational reconstruction to rapidly and robustly estimate a dense depth map from low photon counts. Our sensor fusion approach uses measurements of single photon arrival times from a low-resolution single-photon detector array and an intensity image from a conventional high-resolution camera. Using a multi-scale deep convolutional network, we jointly process the raw measurements from both sensors and output a high-resolution depth map. To demonstrate the efficacy of our approach, we implement a hardware prototype and show results using captured data. At low signal-to-background levels, our depth reconstruction algorithm with sensor fusion outperforms other methods for depth estimation from noisy measurements of photon arrival times.<\/jats:p>","DOI":"10.1145\/3197517.3201316","type":"journal-article","created":{"date-parts":[[2018,7,31]],"date-time":"2018-07-31T15:56:23Z","timestamp":1533052583000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":196,"title":["Single-photon 3D imaging with deep sensor fusion"],"prefix":"10.1145","volume":"37","author":[{"given":"David B.","family":"Lindell","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"O'Toole","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gordon","family":"Wetzstein","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,7,30]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/83.503916"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073686"},{"key":"e_1_2_2_3_1","first-page":"3","article-title":"A Bayesian Approach to Denoising of Single-Photon Binary Images","volume":"3","author":"Altmann Y.","year":"2017","journal-title":"IEEE Trans. Computat. Imaging"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03778-8_5"},{"key":"e_1_2_2_5_1","volume-title":"Proc. SPIE","volume":"9899","author":"Burri S."},{"key":"e_1_2_2_6_1","volume-title":"Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications.","author":"Chan D."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.37"},{"key":"e_1_2_2_8_1","doi-asserted-by":"crossref","unstructured":"H. Dautet P. Deschamps B. Dion A.D. MacGregor D. MacSween R.J. McIntyre C. Trottier and P.P. Webb. 1993. Photon counting techniques with silicon avalanche photodiodes. Applied optics 32 21 (1993) 3894--3900.  H. Dautet P. Deschamps B. Dion A.D. MacGregor D. MacSween R.J. McIntyre C. Trottier and P.P. Webb. 1993. Photon counting techniques with silicon avalanche photodiodes. Applied optics 32 21 (1993) 3894--3900.","DOI":"10.1364\/AO.32.003894"},{"key":"e_1_2_2_9_1","unstructured":"J. Diebel and S. Thrun. 2006. An application of Markov Random Fields to range sensing. In Prac. NIPS. 291--298.   J. Diebel and S. Thrun. 2006. An application of Markov Random Fields to range sensing. In Prac. NIPS. 291--298."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.127"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364911434148"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00138-016-0784-4"},{"key":"e_1_2_2_13_1","volume-title":"Proc. ECCV. 353--369","author":"Hui T."},{"key":"e_1_2_2_14_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma D.","year":"2014"},{"key":"e_1_2_2_15_1","doi-asserted-by":"crossref","unstructured":"A. Kirmani D. Venkatraman D. Shin A. Cola\u00e7o F.N.C. Wong J.H. Shapiro and V.K. Goyal. 2014. First-photon imaging. Science 343 6166 (2014) 58--61.  A. Kirmani D. Venkatraman D. Shin A. Cola\u00e7o F.N.C. Wong J.H. Shapiro and V.K. Goyal. 2014. First-photon imaging. Science 343 6166 (2014) 58--61.","DOI":"10.1126\/science.1246775"},{"key":"e_1_2_2_16_1","unstructured":"A. Kolb E. Barth R. Koch and R. Larsen. 2009. Time-of-flight sensors in computer graphics. In Eurographics (STARs). 119--134.  A. Kolb E. Barth R. Koch and R. Larsen. 2009. Time-of-flight sensors in computer graphics. In Eurographics (STARs). 119--134."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276497"},{"key":"e_1_2_2_18_1","volume-title":"Proc. SPIE 1614","author":"Koskinen M."},{"key":"e_1_2_2_19_1","volume-title":"Proc. ECCV. 154--169","author":"Li Y."},{"key":"e_1_2_2_20_1","volume-title":"Proc. CVPR.","author":"Lin G."},{"key":"e_1_2_2_21_1","volume-title":"Proc. ICCP.","author":"Lindell D.B."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130884"},{"key":"e_1_2_2_23_1","doi-asserted-by":"crossref","unstructured":"A. McCarthy X. Ren A. Della Frera N.R. Gemmell N.J. Krichel C. Scarcella A. Ruggeri A. Tosi and G.S. Buller. 2013. Kilometer-range depth imaging at 1550 nm wavelength using an InGaAs\/InP single-photon avalanche diode detector. Optics express 21 19 (2013) 22098--22113.  A. McCarthy X. Ren A. Della Frera N.R. Gemmell N.J. Krichel C. Scarcella A. Ruggeri A. Tosi and G.S. Buller. 2013. Kilometer-range depth imaging at 1550 nm wavelength using an InGaAs\/InP single-photon avalanche diode detector. Optics express 21 19 (2013) 22098--22113.","DOI":"10.1364\/OE.21.022098"},{"key":"e_1_2_2_24_1","unstructured":"D. O'Connor and D. Philips. 1984. Time-correlated single photon counting. Academic Press.  D. O'Connor and D. Philips. 1984. Time-correlated single photon counting. Academic Press."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766897"},{"key":"e_1_2_2_26_1","volume-title":"Proc. CVPR.","author":"O'Toole M."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2011.6126423"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1364\/OE.25.011919"},{"key":"e_1_2_2_29_1","volume-title":"Proc. CVPR. 1743--1751","author":"Peng C."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015706.1015777"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/jcc.20084"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2017.2706028"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.nima.2006.05.060"},{"key":"e_1_2_2_34_1","volume-title":"Proc. CVPR. 1--8.","author":"Scharstein D."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCI.2015.2453093"},{"key":"e_1_2_2_36_1","doi-asserted-by":"crossref","unstructured":"D. Shin F. Xu D. Venkatraman R. Lussana F. Villa F. Zappa V.K. Goyal F.N.C. Wong and J.H. Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nature Communications 7 (2016).  D. Shin F. Xu D. Venkatraman R. Lussana F. Villa F. Zappa V.K. Goyal F.N.C. Wong and J.H. Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nature Communications 7 (2016).","DOI":"10.1364\/COSI.2016.CW5D.4"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33715-4_54"},{"key":"e_1_2_2_38_1","volume-title":"Proc. CVPR.","author":"Su S."},{"key":"e_1_2_2_39_1","doi-asserted-by":"crossref","unstructured":"R. Tobin A. Halimi A. McCarthy X. Ren K.J. McEwan S. McLaughlin and G.S. Buller 2017. Long-range depth profiling of camouflaged targets using single-photon detection. Optical Engineering 57 (2017).  R. Tobin A. Halimi A. McCarthy X. Ren K.J. McEwan S. McLaughlin and G.S. Buller 2017. Long-range depth profiling of camouflaged targets using single-photon detection. Optical Engineering 57 (2017).","DOI":"10.1117\/1.OE.57.3.031303"},{"key":"e_1_2_2_40_1","volume-title":"Proc. CVPR. 1--8.","author":"Yang Q."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3197517.3201316","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3197517.3201316","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:39:44Z","timestamp":1750210784000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3197517.3201316"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,30]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,8,31]]}},"alternative-id":["10.1145\/3197517.3201316"],"URL":"https:\/\/doi.org\/10.1145\/3197517.3201316","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,7,30]]},"assertion":[{"value":"2018-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}