{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T13:38:30Z","timestamp":1774964310904,"version":"3.50.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2017,11,20]],"date-time":"2017-11-20T00:00:00Z","timestamp":1511136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100010663","name":"H2020 European Research Council","doi-asserted-by":"publisher","award":["CHAMELEON Project, 682080"],"award-info":[{"award-number":["CHAMELEON Project, 682080"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010067","name":"Gobierno de Arag\u00f3n","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100010067","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003329","name":"Ministerio de Econom\u00eda y Competitividad","doi-asserted-by":"publisher","award":["TIN2016-78753-P and TIN2014-61696-EXP"],"award-info":[{"award-number":["TIN2016-78753-P and TIN2014-61696-EXP"]}],"id":[{"id":"10.13039\/501100003329","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2016R1A2B2013031, 2013M3A6A6073718"],"award-info":[{"award-number":["2016R1A2B2013031, 2013M3A6A6073718"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["REVEAL"],"award-info":[{"award-number":["REVEAL"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2017,12,31]]},"abstract":"<jats:p>Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work, we avoid these approaches and propose a new technique to correct errors in depth caused by MPI, which requires no camera modifications and takes just 10 milliseconds per frame. Our observations about the nature of MPI suggest that most of its information is available in image space; this allows us to formulate the depth imaging process as a spatially-varying convolution and use a convolutional neural network to correct MPI errors. Since the input and output data present similar structure, we base our network on an autoencoder, which we train in two stages. First, we use the encoder (convolution filters) to learn a suitable basis to represent MPI-corrupted depth images; then, we train the decoder (deconvolution filters) to correct depth from synthetic scenes, generated by using a physically-based, time-resolved renderer. This approach allows us to tackle a key problem in ToF, the lack of ground-truth data, by using a large-scale captured training set with MPI-corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the decoder stage of the network. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured, incorrect depth as input.<\/jats:p>","DOI":"10.1145\/3130800.3130884","type":"journal-article","created":{"date-parts":[[2017,11,22]],"date-time":"2017-11-22T16:25:08Z","timestamp":1511367908000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":115,"title":["DeepToF"],"prefix":"10.1145","volume":"36","author":[{"given":"Julio","family":"Marco","sequence":"first","affiliation":[{"name":"Universidad de Zaragoza"}]},{"given":"Quercus","family":"Hernandez","sequence":"additional","affiliation":[{"name":"Universidad de Zaragoza"}]},{"given":"Adolfo","family":"Mu\u00f1oz","sequence":"additional","affiliation":[{"name":"Universidad de Zaragoza"}]},{"given":"Yue","family":"Dong","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia"}]},{"given":"Adrian","family":"Jarabo","sequence":"additional","affiliation":[{"name":"Universidad de Zaragoza"}]},{"given":"Min H.","family":"Kim","sequence":"additional","affiliation":[{"name":"KAIST"}]},{"given":"Xin","family":"Tong","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia"}]},{"given":"Diego","family":"Gutierrez","sequence":"additional","affiliation":[{"name":"Universidad de Zaragoza"}]}],"member":"320","published-online":{"date-parts":[[2017,11,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1364\/OL.39.001705"},{"key":"e_1_2_1_2_1","unstructured":"Jean-Yves Bouguet. 2004. Camera calibration toolbox for Matlab. (2004).  Jean-Yves Bouguet. 2004. Camera calibration toolbox for Matlab. (2004)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130800.3130810"},{"key":"e_1_2_1_4_1","first-page":"786404","volume-title":"Proceedings of SPIE","volume":"7864","author":"Dorrington A. A.","unstructured":"A. A. Dorrington , J. P. Godbaz , M. J. Cree , A. D. Payne , and L. V. Streeter . 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras . In Proceedings of SPIE , Vol. 7864 . 786404- 786404 - 786410 . A. A. Dorrington, J. P. Godbaz, M. J. Cree, A. D. Payne, and L. V. Streeter. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proceedings of SPIE, Vol. 7864. 786404-786404-10."},{"key":"e_1_2_1_5_1","first-page":"1","article-title":"Stacked Convolutional Denoising Auto-Encoders for Feature Representation","volume":"99","author":"Du B.","year":"2016","unstructured":"B. Du , W. Xiong , J. Wu , L. Zhang , L. Zhang , and D. Tao . 2016 . Stacked Convolutional Denoising Auto-Encoders for Feature Representation . IEEE Trans. Cybernetics 99 (2016), 1 -- 11 . B. Du, W. Xiong, J. Wu, L. Zhang, L. Zhang, and D. Tao. 2016. Stacked Convolutional Denoising Auto-Encoders for Feature Representation. IEEE Trans. Cybernetics 99 (2016), 1--11.","journal-title":"IEEE Trans. Cybernetics"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.304"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of NIPS.","author":"Eigen David","year":"2014","unstructured":"David Eigen , Christian Puhrsch , and Rob Fergus . 2014 . Depth map prediction from a single image using a multi-scale deep network . In Proceedings of NIPS. David Eigen, Christian Puhrsch, and Rob Fergus. 2014. Depth map prediction from a single image using a multi-scale deep network. In Proceedings of NIPS."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2003.813506"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2015.2421360"},{"key":"e_1_2_1_10_1","volume-title":"SRA: Fast removal of general multipath for ToF sensors","author":"Freedman Daniel","year":"2014","unstructured":"Daniel Freedman , Yoni Smolin , Eyal Krupka , Ido Leichter , and Mirko Schmidt . 2014 . SRA: Fast removal of general multipath for ToF sensors . In Proceedings of ECCV. Springer , 234--249. Daniel Freedman, Yoni Smolin, Eyal Krupka, Ido Leichter, and Mirko Schmidt. 2014. SRA: Fast removal of general multipath for ToF sensors. In Proceedings of ECCV. Springer, 234--249."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2010.874"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39402-7_4"},{"key":"e_1_2_1_13_1","first-page":"829618","volume-title":"Proceedings of SPIE","volume":"8296","author":"Godbaz John P.","unstructured":"John P. Godbaz , Michael J. Cree , and Adrian A. Dorrington . 2012. Closed-form inverses for the mixed pixel\/multipath interference problem in AMCW lidar . In Proceedings of SPIE , Vol. 8296 . 829618- 829618 - 829615 . John P. Godbaz, Michael J. Cree, and Adrian A. Dorrington. 2012. Closed-form inverses for the mixed pixel\/multipath interference problem in AMCW lidar. In Proceedings of SPIE, Vol. 8296. 829618-829618-15."},{"key":"e_1_2_1_14_1","volume-title":"Deep Learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep Learning . MIT Press . http:\/\/www.deeplearningbook.org. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http:\/\/www.deeplearningbook.org."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2735702"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461945"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of CVPR","volume":"1","author":"Huang Jinggang","year":"2000","unstructured":"Jinggang Huang , Ann B Lee , and David Mumford . 2000 . Statistics of range images . In Proceedings of CVPR , Vol. 1 . IEEE, 324--331. Jinggang Huang, Ann B Lee, and David Mumford. 2000. Statistics of range images. In Proceedings of CVPR, Vol. 1. IEEE, 324--331."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of CVPR","volume":"1","author":"Huang Jinggang","year":"1999","unstructured":"Jinggang Huang and David Mumford . 1999 . Statistics of natural images and models . In Proceedings of CVPR , Vol. 1 . IEEE, 541--547. Jinggang Huang and David Mumford. 1999. Statistics of natural images and models. In Proceedings of CVPR, Vol. 1. IEEE, 541--547."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661251"},{"key":"e_1_2_1_21_1","volume-title":"Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective. Visual Informatics 1, 1","author":"Jarabo Adrian","year":"2017","unstructured":"Adrian Jarabo , Belen Masia , Julio Marco , and Diego Gutierrez . 2017. Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective. Visual Informatics 1, 1 ( 2017 ). Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective. Visual Informatics 1, 1 (2017)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2013.10.008"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508363.2508428"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980251"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the IEEE International Conference on Computer Vision Workshops. 1167--1174","author":"Karayev S","year":"2011","unstructured":"S Karayev , Y Jia , J Barron , M Fritz , K Saenko , and T Darrell . 2011 . A category-level 3-D object dataset: putting the Kinect to work . In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1167--1174 . S Karayev, Y Jia, J Barron, M Fritz, K Saenko, and T Darrell. 2011. A category-level 3-D object dataset: putting the Kinect to work. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1167--1174."},{"key":"e_1_2_1_26_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980","author":"Kingma Diederik","year":"2014","unstructured":"Diederik Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980 (2014). Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980 (2014)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICME.2013.6607553"},{"key":"e_1_2_1_28_1","unstructured":"Nick Knighton and Bruce Bugbee. 2005. A mixture of barium sulfate and white paint is a low-cost substitute reflectance standard for Spectralon\u00ae. (2005).  Nick Knighton and Bruce Bugbee. 2005. A mixture of barium sulfate and white paint is a low-cost substitute reflectance standard for Spectralon\u00ae. (2005)."},{"key":"e_1_2_1_29_1","volume-title":"Int. J. of Computer Vision","author":"Lee Ann B","year":"2000","unstructured":"Ann B Lee , JG Huang , and DB Mumford . 2000. Random collage model for natural images . Int. J. of Computer Vision ( 2000 ). Ann B Lee, JG Huang, and DB Mumford. 2000. Random collage model for natural images. Int. J. of Computer Vision (2000)."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of CVPR.","author":"Li Bo","unstructured":"Bo Li , Chunhua Shen , Yuchao Dai , A. van den Hengel, and Mingyi He. 2015. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs . In Proceedings of CVPR. Bo Li, Chunhua Shen, Yuchao Dai, A. van den Hengel, and Mingyi He. 2015. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. In Proceedings of CVPR."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/18.61115"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2009.11.002"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299152"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/2029556.2029563"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298602"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/882262.882280"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601103"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1364\/AO.16.000729"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/1888028.1888085"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2816795.2818103"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1364\/OL.40.000918"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33715-4_54"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of CVPR.","author":"Su Hao","year":"2017","unstructured":"Hao Su , Haoqiang Fan , and Leonidas Guibas . 2017 . A Point Set Generation Network for 3D Object Reconstruction from a Single Image . In Proceedings of CVPR. Hao Su, Haoqiang Fan, and Leonidas Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In Proceedings of CVPR."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461912.2461928"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of CVPR.","author":"Wang Peng","unstructured":"Peng Wang , Xiaohui Shen , Zhe Lin , S. Cohen , B. Price , and A. Yuille . 2015. Towards unified depth and semantic prediction from a single image . In Proceedings of CVPR. Peng Wang, Xiaohui Shen, Zhe Lin, S. Cohen, B. Price, and A. Yuille. 2015. Towards unified depth and semantic prediction from a single image. In Proceedings of CVPR."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-013-0668-2"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.458"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.25"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2010.2050625"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298767"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3130800.3130884","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3130800.3130884","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3130800.3130884","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:18Z","timestamp":1750212678000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3130800.3130884"}},"subtitle":["off-the-shelf real-time correction of multipath interference in time-of-flight imaging"],"short-title":[],"issued":{"date-parts":[[2017,11,20]]},"references-count":50,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2017,12,31]]}},"alternative-id":["10.1145\/3130800.3130884"],"URL":"https:\/\/doi.org\/10.1145\/3130800.3130884","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,11,20]]},"assertion":[{"value":"2017-11-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}