{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T19:56:08Z","timestamp":1777406168585,"version":"3.51.4"},"reference-count":30,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2022,3,22]],"date-time":"2022-03-22T00:00:00Z","timestamp":1647907200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["17H06102"],"award-info":[{"award-number":["17H06102"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["18H05240"],"award-info":[{"award-number":["18H05240"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor.<\/jats:p>","DOI":"10.3390\/s22072442","type":"journal-article","created":{"date-parts":[[2022,3,22]],"date-time":"2022-03-22T23:30:23Z","timestamp":1647991823000},"page":"2442","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor"],"prefix":"10.3390","volume":"22","author":[{"given":"Masaya","family":"Horio","sequence":"first","affiliation":[{"name":"Graduate School of Integrated Science and Technology, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8631-5024","authenticated-orcid":false,"given":"Yu","family":"Feng","sequence":"additional","affiliation":[{"name":"Graduate School of Integrated Science and Technology, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tomoya","family":"Kokado","sequence":"additional","affiliation":[{"name":"Graduate School of Integrated Science and Technology, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taishi","family":"Takasawa","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keita","family":"Yasutomi","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shoji","family":"Kawahito","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Takashi","family":"Komuro","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Engineering, Saitama University, Saitama 338-8570, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hajime","family":"Nagahara","sequence":"additional","affiliation":[{"name":"Institute of Datability Science, Osaka University, Osaka 565-0871, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Keiichiro","family":"Kagawa","sequence":"additional","affiliation":[{"name":"Research Institute of Electronics, Shizuoka University, Shizuoka 422-8011, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Remondino, F., and Stoppa, D. (2013). TOF Range-Imaging Cameras, Springer.","DOI":"10.1007\/978-3-642-27523-4"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"12495","DOI":"10.1109\/ACCESS.2019.2891693","article-title":"Recent Advances in 3D Data Acquisition and Processing by Time-of-Flight Camera","volume":"7","author":"He","year":"2019","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"3419","DOI":"10.1109\/JSEN.2015.2421360","article-title":"Resolving Multi-path Interference in Kinect: An Inverse Problem Approach","volume":"16","author":"Feigin","year":"2016","journal-title":"IEEE Sens. J."},{"key":"ref_4","unstructured":"Whyte, R. (2021, April 28). Multi-Path Interference in Indirect Time-of-Flight Depth Sensors. Available online: https:\/\/medium.com\/chronoptics-time-of-flight\/multi-path-interference-in-indirect-time-of-flight-depth-sensors-7f59c5fcd122."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ximenes, A.R., Padmanabhan, P., Lee, M.-J., Yamashita, Y., Yaung, D.N., and Charbon, E. (2018, January 11\u201315). A 256 \u00d7 256 45\/65nm 3D-Stacked SPAD-Based Direct TOF Image Sensor for LiDAR Applications with Optical Polar Modulation for up to 18.6 dB Interference Suppression. Proceedings of the 2018 IEEE International Solid\u2014State Circuits Conference\u2014(ISSCC), San Francisco, CA, USA.","DOI":"10.1109\/ISSCC.2018.8310201"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1109\/JEDS.2018.2807918","article-title":"Optimization of Pinned Photodiode Pixels for High-Speed Time of Flight Applications","volume":"6","author":"Acerbi","year":"2018","journal-title":"IEEE J. Electron Devices Soc."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1063\/1.1136594","article-title":"Towards picosecond resolution with single-photon avalanche diodes","volume":"52","author":"Cova","year":"1981","journal-title":"Rev. Sci. Instrum."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Bamji, C.S., Mehta, S., Thompson, B., Elkhatib, T., Wurster, S., Akkaya, O., Payne, A., Godbaz, J., Fenton, M., and Rajasekaran, V. (2018, January 11\u201315). IMpixel 65 nm BSI 320MHz Demodulated TOF Image Sensor with 3 \u03bcm Global Shutter Pixels and Analog Binning. Proceedings of the 2018 IEEE International Solid\u2014State Circuits Conference\u2014(ISSCC), San Francisco, CA, USA.","DOI":"10.1109\/ISSCC.2018.8310200"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2291","DOI":"10.1109\/JSSC.2019.2916310","article-title":"A Sub-100 \u03bcm-Range-Resolution Time-of-Flight Range Image Sensor with Three-Tap Lock-In Pixels, Non-Overlapping Gate Clock, and Reference Plane Sampling","volume":"54","author":"Yasutomi","year":"2019","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1145\/2508363.2508428","article-title":"Coded Time of Flight Cameras: Sparse Deconvolution to Address Multi-path Interference and Recover Time Profiles","volume":"32","author":"Kadambi","year":"2013","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/s41074-017-0026-3","article-title":"Recovering Temporal PSF Using ToF Camera with Delayed Light Emission","volume":"9","author":"Kitano","year":"2017","journal-title":"IPSJ Trans. Comput. Vis. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1705","DOI":"10.1364\/OL.39.001705","article-title":"Resolving Multi-path Interference in Time-of-Flight Imaging via Modulation Frequency Diversity and Sparse Regularization","volume":"39","author":"Bhandari","year":"2014","journal-title":"Opt. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Kadambi, A., Schiel, J., and Raskar, R. (2016, January 27\u201330). Macroscopic Interferometry: Rethinking Depth Estimation with Frequency-Domain Time-of-Flight. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.103"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Lickfold, C.A., Streeter, L., Cree, M.J., and Scott, J.B. (2019, January 27\u201330). Simultaneous Phase and Frequency Stepping in Time-of-Flight Range Imaging. Proceedings of the 2019 IEEE SENSORS, Montreal, QC, Canada.","DOI":"10.1109\/SENSORS43011.2019.8956800"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Agresti, G., Schaefer, H., Sartor, P., and Zanuttigh, P. (2019, January 15\u201320). Unsupervised Domain Adaptation for Tof Data Denoising with Adversarial Learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00573"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1145\/3130800.3130884","article-title":"Deeptof: Off-the-Shelf Real-Time Correction of Multi-path Interference in Time-of-Flight Imaging","volume":"36","author":"Marco","year":"2017","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_17","first-page":"202","article-title":"Separation of Multi-Path Components in Sweep-Less Time-of-Flight Depth Imaging with a Temporally-Compressive Multi-Aperture Image Sensor","volume":"6","author":"Mochizuki","year":"2018","journal-title":"ITE Trans. Media Technol. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Mochizuki, F., Kagawa, K., Okihara, S., Seo, M.-W., Zhang, B., Takasawa, T., Yasutomi, K., and Kawahito, S. (2015, January 22\u201326). Single-Shot 200 Mfps 5 \u00d7 3-Aperture Compressive CMOS Imager. Proceedings of the 2015 IEEE International Solid-State Circuits Conference-(ISSCC) Digest of Technical Papers, San Francisco, CA, USA.","DOI":"10.1109\/ISSCC.2015.7062953"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1109\/MSP.2007.4286571","article-title":"Compressive Sensing [Lecture Notes]","volume":"24","author":"Baraniuk","year":"2007","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/MSP.2007.914729","article-title":"Imaging via Compressive Sampling","volume":"25","author":"Romberg","year":"2008","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"4961","DOI":"10.3390\/s130404961","article-title":"Compressive Sensing Image Sensors-Hardware Implementation","volume":"13","author":"Dadkhah","year":"2013","journal-title":"Sensors"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/MSP.2007.914730","article-title":"Single-Pixel Imaging via Compressive Sampling","volume":"25","author":"Duarte","year":"2008","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_23","unstructured":"Kagawa, K., Kokado, T., Sato, Y., Mochizuki, F., Nagahara, H., Takasawa, T., Yasutomi, K., and Kawahito, S. (2019, January 23\u201327). Multi-Tap Macro-Pixel Based Comporessive Ultra-High-Speed CMOS Image Sensor. Proceedings of the 2019 International Image Sensor Workshop, Snowbird, UT, USA."},{"key":"ref_24","unstructured":"Li, C., Yin, W., and Zhang, Y. (2021, April 28). TVAL3: TV Minimization by Augmented Lagrangian and Alternating Direction Algorithms. Available online: https:\/\/www.caam.rice.edu\/~optimization\/L1\/TVAL3\/."},{"key":"ref_25","unstructured":"Li, C. (2010). An Efficient Algorithm for Total Variation Regularization with Applications to the Single Pixel Camera and Compressive Sensing. [Ph.D. Thesis, Rice University]."},{"key":"ref_26","unstructured":"D\u2019Errico, J. (2021, April 28). Fminsearchbnd, Fminsearchcon. Available online: https:\/\/jp.mathworks.com\/matlabcentral\/fileexchange\/8277-fminsearchbnd-fminsearchcon."},{"key":"ref_27","unstructured":"Kawahito, S., Baek, G., Li, Z., Han, S.-M., Seo, M.-W., Yasutomi, K., and Kagawa, K. (2013, January 12\u201316). CMOS Lock-in Pixel Image Sensors with Lateral Electric Field Control for Time-Resolved Imaging. Proceedings of the International Image Sensor Workshop (IISW), Snowbird, UT, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/JSSC.2015.2496788","article-title":"A 10 Ps Time-Resolution CMOS Image Sensor with Two-Tap True-CDS Lock-in Pixels for Fluorescence Lifetime Imaging","volume":"51","author":"Seo","year":"2015","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lee, S., Yasutomi, K., Morita, M., Kawanishi, H., and Kawahito, S. (2020). A Time-of-Flight Range Sensor Using Four-Tap Lock-In Pixels with High near Infrared Sensitivity for LiDAR Applications. Sensors, 20.","DOI":"10.3390\/s20010116"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Conde, M.H., Kagawa, K., Kokado, T., Kawahito, S., and Loffeld, O. (2020, January 4\u20138). Single-Shot Real-Time Multiple-Path Time-of-Flight Depth Imaging for Multi-Aperture and Macro-Pixel Sensors. Proceedings of the ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9054565"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/7\/2442\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:41:01Z","timestamp":1760136061000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/7\/2442"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,22]]},"references-count":30,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["s22072442"],"URL":"https:\/\/doi.org\/10.3390\/s22072442","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,22]]}}}