{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:59:46Z","timestamp":1760234386385,"version":"build-2065373602"},"reference-count":45,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T00:00:00Z","timestamp":1619568000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["2020GFYD011","2020 GFZD008"],"award-info":[{"award-number":["2020GFYD011","2020 GFZD008"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Depth maps obtained through sensors are often unsatisfactory because of their low-resolution and noise interference. In this paper, we propose a real-time depth map enhancement system based on a residual network which uses dual channels to process depth maps and intensity maps respectively and cancels the preprocessing process, and the algorithm proposed can achieve real-time processing speed at more than 30 fps. Furthermore, the FPGA design and implementation for depth sensing is also introduced. In this FPGA design, intensity image and depth image are captured by the dual-camera synchronous acquisition system as the input of neural network. Experiments on various depth map restoration shows our algorithms has better performance than existing LRMC, DE-CNN and DDTF algorithms on standard datasets and has a better depth map super-resolution, and our FPGA completed the test of the system to ensure that the data throughput of the USB 3.0 interface of the acquisition system is stable at 226 Mbps, and support dual-camera to work at full speed, that is, 54 fps@ (1280 \u00d7 960 + 328 \u00d7 248 \u00d7 3).<\/jats:p>","DOI":"10.3390\/e23050546","type":"journal-article","created":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T10:41:58Z","timestamp":1619606518000},"page":"546","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Residual Network and FPGA Based Real-Time Depth Map Enhancement System"],"prefix":"10.3390","volume":"23","author":[{"given":"Zhenni","family":"Li","sequence":"first","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Haoyi","family":"Sun","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuliang","family":"Gao","sequence":"additional","affiliation":[{"name":"College of Artificial Intelligence, Nankai University, Tianjin 300071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiao","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Northeastern University, Shenyang 110819, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1109\/TIP.2007.914755","article-title":"Vision processing for realtime 3-D data acquisition based on coded structured light","volume":"17","author":"Chen","year":"2008","journal-title":"IEEE Trans. Image Process."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Li, L., Xiang, S., Yang, Y., and Yu, L. (2015, January 27\u201330). Multi-camera interference cancellation of time-of-flight (TOF) cameras. Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada.","DOI":"10.1109\/ICIP.2015.7350860"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hou, Y., Chiou, S., and Lin, M. (2017, January 1\u20133). Real-time detection and tracking for moving objects based on computer vision method. Proceedings of the 2017 2nd International Conference on Control. and Robotics Engineering (ICCRE), Bangkok, Thailand.","DOI":"10.1109\/ICCRE.2017.7935072"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"836","DOI":"10.1109\/TIP.2016.2621673","article-title":"Exploiting depth from single monocular images for object detection and semantic segmentation","volume":"26","author":"Cao","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Raghunandan, A., Raghav, P., and Aradhya, H.V.R. (2018, January 3\u20135). Object detection algorithms for video surveillance applications. Proceedings of the 2018 International Conference on Communication and Signal Processing (ICCSP), Chennai, India.","DOI":"10.1109\/ICCSP.2018.8524461"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1109\/TCYB.2016.2638856","article-title":"Joint-feature guided depth map super-resolution with face priors","volume":"48","author":"Yang","year":"2018","journal-title":"IEEE Trans. Cybern."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Xu, D., Fan, X., Zhao, D., and Gao, W. (2018, January 7\u201310). Multiresolution contourlet transform fusion based depth map super resolution. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451042"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Li, W., Hu, W., Dong, T., and Qu, J. (2018, January 8\u20139). Depth image enhancement algorithm based on RGB image fusion. Proceedings of the 2018 11th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China.","DOI":"10.1109\/ISCID.2018.10126"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"772","DOI":"10.1109\/TMM.2020.2987706","article-title":"Frequency-dependent depth map enhancement via iterative depth-guided affine transformation and intensity-guided refinement","volume":"23","author":"Zuo","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_10","unstructured":"Si, L., Xiaofeng, R., and Feng, L. (2014, January 24\u201327). Depth enhancement via low-rank Matrix completion. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA."},{"key":"ref_11","unstructured":"Xin, Z., and Ruiyuan, W. (2016, January 20\u201325). Fast depth image denoising and enhancement using a deep convolutional network. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, Shanghai, China."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s40305-015-0074-2","article-title":"Data-driven tight frame for multi-channel images and its application to joint color-depth image reconstruction","volume":"3","author":"Wang","year":"2015","journal-title":"J. Oper. Res. Soc. China"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"26666","DOI":"10.1109\/ACCESS.2017.2773141","article-title":"Color-guided depth map super resolution using convolutional neural network","volume":"5","author":"Ni","year":"2017","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhou, W., Li, X., and Reynolds, D. (2017, January 5\u20139). Guided deep network for depth map super-resolution: How much can color help?. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.","DOI":"10.1109\/ICASSP.2017.7952398"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Chen, B., and Jung, C. (2018, January 15\u201320). Single depth image super-resolution using convolutional neural networks. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8462043"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Korinevskaya, A., and Makarov, I. (2018, January 16\u201320). Fast depth map super-resolution using deep neural network. Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Munich, Germany.","DOI":"10.1109\/ISMAR-Adjunct.2018.00047"},{"key":"ref_17","unstructured":"Li, B., Dai, Y., Chen, H., and He, M. (2017). Single image depth estimation by dilated deep residual convolutional neural network and soft-weight-sum inference. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"4676","DOI":"10.1109\/TIP.2018.2832296","article-title":"Learning depth from single images with deep neural network embedding focal length","volume":"27","author":"He","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kumari, S., Jha, R.R., Bhavsar, A., and Nigam, A. (2019, January 22\u201325). AUTODEPTH: Single image depth map estimation via residual CNN encoder-decoder and stacked hourglass. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803006"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Siddiqui, S.A., Vierling, A., and Berns, K. (2020). Multi-modal depth estimation using convolutional neural networks. arXiv.","DOI":"10.1109\/SSRR50563.2020.9292608"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1109\/TMI.2017.2760978","article-title":"A deep cascade of convolutional neural networks for dynamic MR image reconstruction","volume":"37","author":"Schlemper","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Li, Z., Liu, X., Creighton, F.X., Taylor, R.H., and Unberath, M. (2020). Revisiting stereo depth estimation from a sequence-to-sequence perspective with transformers. arXiv.","DOI":"10.1109\/ICCV48922.2021.00614"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Qian, T., Chen, L., Li, X., Sun, H., and Ni, J. (2018, January 24\u201326). A 1.25 Gbps programmable FPGA I\/O buffer with multi-standard support. Proceedings of the 2018 IEEE 3rd International Conference on Integrated Circuits and Microsystems (ICICM), Shanghai, China.","DOI":"10.1109\/ICAM.2018.8596436"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1109\/TNNLS.2018.2844093","article-title":"FpgaConvNet: Mapping regular and irregular convolutional neural networks on FPGAs","volume":"30","author":"Venieris","year":"2019","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ahmad, A., and Pasha, M.A. (2019, January 25\u201329). Towards design space exploration and optimization of fast algorithms for convolutional neural networks (CNNs) on FPGAs. Proceedings of the 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), Florence, Italy.","DOI":"10.23919\/DATE.2019.8715272"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Jiang, Y., and Luo, J. (2020, January 14\u201316). Target height measurement method based on stereo vision. Proceedings of the 2020 International Conference on Information Science, Parallel and Distributed Systems (ISPDS), Xi\u2019an, China.","DOI":"10.1109\/ISPDS51347.2020.00011"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Deng, H., Dong, P., Li, Z., Lyu, H., Zhang, Y., Luo, Y., and An, F. (2020, January 23\u201325). Robot navigation based on pseudo-binocular stereo vision and linear fitting. Proceedings of the 2020 IEEE International Conference on Integrated Circuits, Technologies and Applications (ICTA), Nanjing, China.","DOI":"10.1109\/ICTA50426.2020.9332014"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Dong, H., Zhang, Y., Chen, M., and Jin, W. (2017, January 21\u201323). Design of the image acquisition and processing system for color sorter based on FPGA. Proceedings of the 2017 2nd International Conference on Cybernetics, Robotics and Control (CRC), Chengdu, China.","DOI":"10.1109\/CRC.2017.43"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Manabe, T., Shibata, Y., and Oguri, K. (2017, January 11\u201313). FPGA implementation of a real-time super-resolution system with a CNN based on a residue number system. Proceedings of the 2017 International Conference on Field Programmable Technology (ICFPT), Melbourne, VIC, Australia.","DOI":"10.1109\/FPT.2017.8280165"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Shandilya, R., and Sharma, R.K. (2017, January 11\u201312). FPGA implementation of image enhancement technique for Automatic Vehicles Number Plate detection. Proceedings of the 2017 International Conference on Trends in Electronics and Informatics (ICEI), Tirunelveli, India.","DOI":"10.1109\/ICOEI.2017.8300860"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Prashant, G.P., and Jagdale, S.M. (2017, January 5\u20136). Information fusion for images on FPGA: Pixel level with pseudo color. Proceedings of the 2017 1st International Conference on Intelligent Systems and Information Management (ICISIM), Aurangabad, India.","DOI":"10.1109\/ICISIM.2017.8122171"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Pfeifer, M., Scholl, P.M., Voigt, R., and Becker, B. (May, January 28). Active stereo vision with high resolution on an FPGA. Proceedings of the 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), San Diego, CA, USA.","DOI":"10.1109\/FCCM.2019.00026"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Lee, Y., Choi, S., Lee, E., Lee, S., and Jang, S. (2019, January 15\u201318). A real-time AD-census stereo matching based on FPGA. Proceedings of the 2019 19th International Conference on Control, Automation and Systems (ICCAS), Jeju, Korea.","DOI":"10.23919\/ICCAS47443.2019.8971538"},{"key":"ref_34","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2016, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_36","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very deep convolutional networks for large-scale image recognition. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Butler, D.J., Wulff, J., Stanley, G.B., and Black, M.J. (2012, January 7\u201313). A naturalistic open source movie for optical flow evaluation. Proceedings of the European Conference on Computer Vision, Florence, Italy.","DOI":"10.1007\/978-3-642-33783-3_44"},{"key":"ref_38","unstructured":"Kingma, D.P., and Ba, J.L.A. (2017). A method for Stochastic optimization. arXiv."},{"key":"ref_39","unstructured":"Cypress (2020, December 16). EZ-USB FX3 SuperSpeed USB Controller 2012. Available online: www.cypress.com."},{"key":"ref_40","unstructured":"Xilinx (2020, December 16). AXI IIC Bus Interface LogiCORE IP Product Guide (PG090). Available online: https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/axi_iic\/v1_02_a\/pg090-axi-iic.pdf."},{"key":"ref_41","unstructured":"Xilinx (2020, December 16). UltraScale Architecture Libraries Guide (UG974). Available online: www.xilinx.com."},{"key":"ref_42","unstructured":"Xilinx (2020, December 16). FIFO Generator LogiCORE IP Product Guide (PG057). Available online: https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/fifo_generator\/v9_3\/pg057-fifo-generator.pdf."},{"key":"ref_43","unstructured":"Cypress (2020, December 16). Designing with the EZ-USB FX3 Slave FIFO Interface (AN75705). Available online: www.cypress.com."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1109\/TIP.2016.2621410","article-title":"Depth map restoration from under-sampled data","volume":"26","author":"Mandal","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_45","unstructured":"Aodha, O.M., Campbell, N.D.F., Nair, A., and Brostow, G.J. (2012, January 7\u201313). Patch based synthesis for single depth image super-resolution. Proceedings of the 2012 Springer European Conference on Computer Vision, Florence, Italy."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/5\/546\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:54:38Z","timestamp":1760162078000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/5\/546"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,28]]},"references-count":45,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["e23050546"],"URL":"https:\/\/doi.org\/10.3390\/e23050546","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2021,4,28]]}}}