{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,18]],"date-time":"2026-04-18T18:44:28Z","timestamp":1776537868811,"version":"3.51.2"},"reference-count":48,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T00:00:00Z","timestamp":1615420800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this work, we propose a novel approach for correcting multi-path interference (MPI) in Time-of-Flight (ToF) cameras by estimating the direct and global components of the incoming light. MPI is an error source linked to the multiple reflections of light inside a scene; each sensor pixel receives information coming from different light paths which generally leads to an overestimation of the depth. We introduce a novel deep learning approach, which estimates the structure of the time-dependent scene impulse response and from it recovers a depth image with a reduced amount of MPI. The model consists of two main blocks: a predictive model that learns a compact encoded representation of the backscattering vector from the noisy input data and a fixed backscattering model which translates the encoded representation into the high dimensional light response. Experimental results on real data show the effectiveness of the proposed approach, which reaches state-of-the-art performances.<\/jats:p>","DOI":"10.3390\/s21061962","type":"journal-article","created":{"date-parts":[[2021,3,11]],"date-time":"2021-03-11T05:38:22Z","timestamp":1615441102000},"page":"1962","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Deep Learning for Transient Image Reconstruction from ToF Data"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3876-2380","authenticated-orcid":false,"given":"Enrico","family":"Buratto","sequence":"first","affiliation":[{"name":"Department of Information Engineering, University of Padova, Via Gradenigo 6\/b, 35131 Padova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adriano","family":"Simonetto","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Padova, Via Gradenigo 6\/b, 35131 Padova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianluca","family":"Agresti","sequence":"additional","affiliation":[{"name":"R&amp;D Center Europe Stuttgart Laboratory 1, Sony Europe B.V., Hedelfinger Str. 61, 70327 Stuttgart, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henrik","family":"Sch\u00e4fer","sequence":"additional","affiliation":[{"name":"R&amp;D Center Europe Stuttgart Laboratory 1, Sony Europe B.V., Hedelfinger Str. 61, 70327 Stuttgart, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9502-2389","authenticated-orcid":false,"given":"Pietro","family":"Zanuttigh","sequence":"additional","affiliation":[{"name":"Department of Information Engineering, University of Padova, Via Gradenigo 6\/b, 35131 Padova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"14991","DOI":"10.1007\/s11042-015-2451-6","article-title":"Hand gesture recognition with jointly calibrated leap motion and depth sensor","volume":"75","author":"Marin","year":"2016","journal-title":"Multimed. Tools Appl."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1109\/MRA.2006.1678144","article-title":"Simultaneous localization and mapping: Part I","volume":"13","author":"Bailey","year":"2006","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1117\/1.1631921","article-title":"Review of 20 years of range sensor development","volume":"13","author":"Blais","year":"2004","journal-title":"J. Electron. Imaging"},{"key":"ref_4","unstructured":"Kim, Y.M., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., and Thrun, S. (October, January 27). Multi-view image and ToF sensor fusion for dense 3D reconstruction. Proceedings of the International Conference on Computer Vision Workshops (ICCVW), Kyoto, Japan."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kerl, C., Souiai, M., Sturm, J., and Cremers, D. (2014, January 8\u201311). Towards Illumination-Invariant 3D Reconstruction Using ToF RGB-D Cameras. Proceedings of the 2014 2nd International Conference on 3D Vision, Tokyo, Japan.","DOI":"10.1109\/3DV.2014.62"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1236021","DOI":"10.1155\/2020\/1236021","article-title":"Vision-Based Three-Dimensional Reconstruction and Monitoring of Large-Scale Steel Tubular Structures","volume":"2020","author":"Tang","year":"2020","journal-title":"Adv. Civ. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.rcim.2019.03.001","article-title":"Real-time detection of surface deformation and strain in recycled aggregate concrete-filled steel tubular columns via four-ocular vision","volume":"59","author":"Tang","year":"2019","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zhu, Q., Chen, L., Li, Q., Li, M., N\u00fcchter, A., and Wang, J. (2012, January 3\u20137). 3D LIDAR point cloud based intersection recognition for autonomous driving. Proceedings of the 2012 IEEE Intelligent Vehicles Symposium, Madrid, Spain.","DOI":"10.1109\/IVS.2012.6232219"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, Y., Chao, W.L., Garg, D., Hariharan, B., Campbell, M., and Weinberger, K.Q. (2019, January 15\u201320). Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00864"},{"key":"ref_10","first-page":"27","article-title":"Lidar systems for precision navigation and safe landing on planetary bodies","volume":"Volume 8192","author":"Amzajerdian","year":"2011","journal-title":"International Symposium on Photoelectronic Detection and Imaging 2011: Laser Sensing and Imaging; and Biological and Medical Applications of Photonics Sensing and Imaging"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1489","DOI":"10.1109\/21.44067","article-title":"Structure from stereo-a review","volume":"19","author":"Dhond","year":"1989","journal-title":"IEEE Trans. Syst. Man, Cybern."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zanuttigh, P., Marin, G., Dal Mutto, C., Dominio, F., Minto, L., and Cortelazzo, G.M. (2016). Time-of-Flight and Structured Light Depth Cameras: Technology and Applications, Springer.","DOI":"10.1007\/978-3-319-30973-6"},{"key":"ref_13","first-page":"44","article-title":"Lidar remote sensing for forestry","volume":"98","author":"Dubayah","year":"2000","journal-title":"J. For."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1005","DOI":"10.1007\/s00138-016-0784-4","article-title":"An Overview of Depth Cameras and Range Scanners Based on Time-of-Flight Technologies","volume":"27","author":"Horaud","year":"2016","journal-title":"Mach. Vis. Appl."},{"key":"ref_15","first-page":"1","article-title":"Theoretical and experimental error analysis of continuous-wave time-of-flight range cameras","volume":"48","author":"Frank","year":"2009","journal-title":"Opt. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2735702","article-title":"Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging","volume":"34","author":"Gupta","year":"2015","journal-title":"ACM Trans. Graph."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Su, S., Heide, F., Wetzstein, G., and Heidrich, W. (2018, January 18\u201322). Deep End-to-End Time-of-Flight Imaging. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00668"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kilho, S., Liu, M., and Taguchi, Y. (2016, January 16\u201321). Learning to remove multipath distortions in Time-of-Flight range images for a robotic arm setup. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.","DOI":"10.1109\/ICRA.2016.7487515"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3130800.3130884","article-title":"DeepToF: Off-the-Shelf Real-Time Correction of Multipath Interference in Time-of-Flight Imaging","volume":"36","author":"Marco","year":"2017","journal-title":"ACM Trans. Graph."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Agresti, G., Schaefer, H., Sartor, P., and Zanuttigh, P. (2019, January 15\u201320). Unsupervised Domain Adaptation for ToF Data Denoising with Adversarial Learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00573"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lefloch, D., Nair, R., Lenzen, F., Sch\u00e4fer, H., Streeter, L., Cree, M.J., Koch, R., and Kolb, A. (2013). Technical foundation and calibration methods for time-of-flight cameras. Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, Springer.","DOI":"10.1007\/978-3-642-44964-2_1"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1318","DOI":"10.1016\/j.cviu.2009.11.002","article-title":"Time-of-flight sensor calibration for accurate range sensing","volume":"114","author":"Lindner","year":"2010","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lenzen, F., Sch\u00e4fer, H., and Garbe, C. (2011). Denoising time-of-flight data with adaptive total variation. International Symposium on Visual Computing, Springer.","DOI":"10.1007\/978-3-642-24028-7_31"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Agresti, G., and Zanuttigh, P. (2018, January 8\u201314). Deep Learning for Multi-Path Error Removal in ToF Sensors. Proceedings of the European Conference on Computer Vision Workshops (ECCVW), Munich, Germany.","DOI":"10.1007\/978-3-030-11015-4_30"},{"key":"ref_25","first-page":"65","article-title":"Recent advances in transient imaging: A computer graphics and vision perspective","volume":"1","author":"Jarabo","year":"2017","journal-title":"Vis. Inform."},{"key":"ref_26","unstructured":"Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. (October, January 29). Looking around the corner using transient imaging. Proceedings of the International Conference on Computer Vision (ICCV), Kyoto, Japan."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sun, Q., Dun, X., Peng, Y., and Heidrich, W. (2018, January 18\u201322). Depth and Transient Imaging with Compressive SPAD Array Cameras. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00036"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2661229.2661260","article-title":"Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport","volume":"33","author":"Heide","year":"2014","journal-title":"ACM Trans. Graph."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1038\/nature25489","article-title":"Confocal non-line-of-sight imaging based on the light-cone transform","volume":"555","author":"Lindell","year":"2018","journal-title":"Nature"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Xin, S., Nousias, S., Kutulakos, K.N., Sankaranarayanan, A.C., Narasimhan, S.G., and Gkioulekas, I. (2019, January 15\u201320). A Theory of Fermat Paths for Non-Line-Of-Sight Shape Reconstruction. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00696"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1038\/s41586-019-1461-3","article-title":"Non-line-of-sight imaging using phasor-field virtual wave optics","volume":"572","author":"Liu","year":"2019","journal-title":"Nature"},{"key":"ref_32","unstructured":"Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020). Spatial Hierarchy Aware Residual Pyramid Network for Time-of-Flight Depth Denoising. Proceedings of the European Conference on Computer Vision (ECCV), Springer International Publishing."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fuchs, S. (2010, January 23\u201326). Multipath Interference Compensation in Time-of-Flight Camera Images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.874"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, M., Leibe, B., and Neumann, B. (2013). Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration. Computer Vision Systems, Springer.","DOI":"10.1007\/978-3-642-39402-7"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.imavis.2013.10.008","article-title":"Modeling and correction of multipath interference in time of flight cameras","volume":"32","author":"Pizarro","year":"2014","journal-title":"Image Vis. Comput."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). SRA: Fast Removal of General Multipath for ToF Sensors. Proceedings of the European Conference on Computer Vision (ECCV), Springer International Publishing.","DOI":"10.1007\/978-3-319-10602-1"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bhandari, A., Feigin, M., Izadi, S., Rhemann, C., Schmidt, M., and Raskar, R. (2014, January 2\u20135). Resolving multipath interference in Kinect: An inverse problem approach. Proceedings of the SENSORS, 2014 IEEE, Valencia, Spain.","DOI":"10.1109\/ICSENS.2014.6985073"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Guo, Q., Frosio, I., Gallo, O., Zickler, T., and Kautz, J. (2018, January 8\u201314). Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01246-5_23"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Marin, G., Zanuttigh, P., and Mattoccia, S. (2016). Reliable fusion of tof and stereo depth driven by confidence measures. European Conference on Computer Vision (ECCV), Springer.","DOI":"10.1007\/978-3-319-46478-7_24"},{"key":"ref_40","first-page":"425","article-title":"Fusion of stereo vision and time-of-flight imaging for improved 3d estimation","volume":"5","author":"Gudmundsson","year":"2008","journal-title":"Int. J. Intell. Syst. Technol. Appl."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1411","DOI":"10.1109\/JSEN.2019.2946591","article-title":"Confidence Estimation for ToF and Stereo Sensors and Its Application to Depth Data Fusion","volume":"20","author":"Poggi","year":"2020","journal-title":"IEEE Sensors J."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1117\/1.OE.54.11.113109","article-title":"Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods","volume":"54","author":"Whyte","year":"2015","journal-title":"Opt. Eng."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Agresti, G., and Zanuttigh, P. (2018, January 8\u201314). Combination of Spatially-Modulated ToF and Structured Light for MPI-Free Depth Estimation. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany.","DOI":"10.1007\/978-3-030-11009-3_21"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., and Bing Kang, S. (2015, January 7\u201312). A Light Transport Model for Mitigating Multipath Interference in Time-of-Flight Sensors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298602"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Jongenelen, A.P.P., Carnegie, D.A., Payne, A.D., and Dorrington, A.A. (2010, January 3\u20136). Maximizing precision over extended unambiguous range for TOF range imaging systems. Proceedings of the 2010 IEEE Instrumentation Measurement Technology Conference Proceedings, Austin, TX, USA.","DOI":"10.1109\/IMTC.2010.5488178"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1221","DOI":"10.1007\/s10044-018-0697-0","article-title":"Training neural networks on high-dimensional data using random projection","volume":"22","author":"Kurdziel","year":"2019","journal-title":"Pattern Anal. Appl."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Liu, B., Wei, Y., Zhang, Y., and Yang, Q. (2017, January 19\u201325). Deep Neural Networks for High Dimension, Low Sample Size Data. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/318"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"786404","DOI":"10.1117\/12.876586","article-title":"Separating true range measurements from multi-path and scattering interference in commercial range cameras","volume":"Volume 7864","author":"Dorrington","year":"2011","journal-title":"Three-Dimensional Imaging, Interaction, and Measurement"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1962\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:33:57Z","timestamp":1760160837000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/6\/1962"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,11]]},"references-count":48,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,3]]}},"alternative-id":["s21061962"],"URL":"https:\/\/doi.org\/10.3390\/s21061962","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,11]]}}}