{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:24:19Z","timestamp":1760243059910,"version":"build-2065373602"},"reference-count":56,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2015,8,21]],"date-time":"2015-08-21T00:00:00Z","timestamp":1440115200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61300131"],"award-info":[{"award-number":["61300131"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Key Technology Research and Development Program of China","award":["2013BAK03B07"],"award-info":[{"award-number":["2013BAK03B07"]}]},{"name":"National High Technology Research and Development Program of China (863 Program)","award":["2013AA013902"],"award-info":[{"award-number":["2013AA013902"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Depth estimation is a classical problem in computer vision, which typically relies on either a depth sensor or stereo matching alone. The depth sensor provides real-time estimates in repetitive and textureless regions where stereo matching is not effective. However, stereo matching can obtain more accurate results in rich texture regions and object boundaries where the depth sensor often fails. We fuse stereo matching and the depth sensor using their complementary characteristics to improve the depth estimation. Here, texture information is incorporated as a constraint to restrict the pixel\u2019s scope of potential disparities and to reduce noise in repetitive and textureless regions. Furthermore, a novel pseudo-two-layer model is used to represent the relationship between disparities in different pixels and segments. It is more robust to luminance variation by treating information obtained from a depth sensor as prior knowledge. Segmentation is viewed as a soft constraint to reduce ambiguities caused by under- or over-segmentation. Compared to the average error rate 3.27% of the previous state-of-the-art methods, our method provides an average error rate of 2.61% on the Middlebury datasets, which shows that our method performs almost 20% better than other \u201cfused\u201d algorithms in the aspect of precision.<\/jats:p>","DOI":"10.3390\/s150820894","type":"journal-article","created":{"date-parts":[[2015,8,21]],"date-time":"2015-08-21T10:38:09Z","timestamp":1440153489000},"page":"20894-20924","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Reliable Fusion of Stereo Matching and Depth Sensor for High Quality Dense Depth Maps"],"prefix":"10.3390","volume":"15","author":[{"given":"Jing","family":"Liu","sequence":"first","affiliation":[{"name":"The Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute ofComputing Technology, Chinese Academy of Sciences, No.6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190, China"},{"name":"University of Chinese Academy of Sciences, No.19A Yuquan Road, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunpeng","family":"Li","sequence":"additional","affiliation":[{"name":"The Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute ofComputing Technology, Chinese Academy of Sciences, No.6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuefeng","family":"Fan","sequence":"additional","affiliation":[{"name":"The Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute ofComputing Technology, Chinese Academy of Sciences, No.6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190, China"},{"name":"University of Chinese Academy of Sciences, No.19A Yuquan Road, Beijing 100049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhaoqi","family":"Wang","sequence":"additional","affiliation":[{"name":"The Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute ofComputing Technology, Chinese Academy of Sciences, No.6 Kexueyuan South Road Zhongguancun, Haidian District, Beijing 100190, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2015,8,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"3491","DOI":"10.3390\/s150203491","article-title":"3D modeling of building indoor spaces and closed doors from imagery and point clouds","volume":"15","author":"Khoshelham","year":"2015","journal-title":"Sensors"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8232","DOI":"10.3390\/s150408232","article-title":"A kinect-based real-time compressive tracking prototype system for amphibious spherical robots","volume":"15","author":"Pan","year":"2015","journal-title":"Sensors"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9228","DOI":"10.3390\/s150409228","article-title":"Visual object recognition with 3D-aware features in KITTI urban scenes","volume":"15","author":"Yebes","year":"2015","journal-title":"Sensors"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/MSP.2010.939077","article-title":"Free-viewpoint TV","volume":"28","author":"Tanimoto","year":"2011","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_5","first-page":"1","article-title":"3D entity-based stereo matching with ground control points and joint second-order smoothness prior","volume":"31","author":"Liu","year":"2014","journal-title":"Vis. Comput."},{"key":"ref_6","unstructured":"ASUS Xtion. Available online: www.asus.com\/Multimedia\/Xtion\/."},{"key":"ref_7","unstructured":"Microsoft Kinect. Available online: www.microsoft.com\/zh-cn\/kinectforwindows\/."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"855","DOI":"10.1007\/s00371-014-0965-y","article-title":"Estimation of kinect depth confidence through self-training","volume":"30","author":"Song","year":"2014","journal-title":"Vis. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1023\/A:1014573219977","article-title":"A taxonomy and evaluation of dense two-frame stereo correspondence algorithms","volume":"47","author":"Scharstein","year":"2002","journal-title":"Int. J. Comput. Vis."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"650","DOI":"10.1109\/TPAMI.2006.70","article-title":"Adaptive support-weight approach for correspondence search","volume":"28","author":"Yoon","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2115","DOI":"10.1109\/TPAMI.2009.131","article-title":"Global stereo reconstruction under second-order smoothness priors","volume":"31","author":"Woodford","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","first-page":"1","article-title":"Patch match stereo-stereo matching with slanted support windows","volume":"11","author":"Bleyer","year":"2011","journal-title":"BMVC"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Mei, X., Sun, X., Dong, W., Wang, H., and Zhang, X. (2013, January 23\u201328). Segment-tree based Cost Aggregation for Stereo Matching. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.47"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, L., and Yang, R. (2011, January 20\u201325). Global Stereo Matching Leveraged by Sparse Ground Control Points. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2011.5995480"},{"key":"ref_15","unstructured":"LiDAR. Available online: www.lidarusa.com."},{"key":"ref_16","unstructured":"3DV Systems. Available online: www.3dvsystems.com."},{"key":"ref_17","unstructured":"Photonix Mixer Device for Distance Measurement. Available online: www.pmdtec.com."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.3390\/s120201437","article-title":"Accuracy and resolution of kinect depth data for indoor mapping applications","volume":"12","author":"Khoshelham","year":"2012","journal-title":"Sensors"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1109\/TPAMI.2009.68","article-title":"Spatial-temporal fusion for high accuracy depth maps using dynamic MRFs","volume":"32","author":"Zhu","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1400","DOI":"10.1109\/TPAMI.2010.172","article-title":"Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps","volume":"33","author":"Zhu","year":"2011","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yang, Q., Tan, K.H., Culbertson, B., and Apostolopoulos, J. (2010, January 4\u20136). Fusion of Active and Passive Sensors for Fast 3D Capture. Proceedings of the IEEE International Workshop on Multimedia Signal Processing, Saint Malo, France.","DOI":"10.1109\/MMSP.2010.5661996"},{"key":"ref_22","unstructured":"Zhang, S., Wang, C., and Chan, S. (2013, January 8\u201310). A New High Resolution Depth Map Estimation System Using Stereo Vision and Depth Sensing Device. Proceedings of the IEEE 9th International Colloquium on Signal Processing and its Applications, Kuala Lumpur, Malaysia."},{"key":"ref_23","first-page":"109","article-title":"A fusion framework of stereo vision and Kinect for high-quality dense depth maps","volume":"7729","author":"Wang","year":"2013","journal-title":"Comput. Vis."},{"key":"ref_24","unstructured":"Somanath, G., Cohen, S., Price, B., and Kambhamettu, C. (July, January 29). Stereo Kinect for High Resolution Stereo Correspondences. Proceedings of the IEEE International Conference on 3DTV-Conference, Seattle, Washington, DC, USA."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1109\/34.888718","article-title":"A flexible new technique for camera calibration","volume":"22","author":"Zhang","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2058","DOI":"10.1109\/TPAMI.2012.125","article-title":"Joint depth and color camera calibration with distortion correction","volume":"34","author":"Herrera","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","unstructured":"Yang, Q. (2012, January 16\u201321). A Non-Local Cost Aggregation Method for Stereo Matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1109\/ICPR.2002.1047421","article-title":"Synergism in low level vision","volume":"4","author":"Christoudias","year":"2002","journal-title":"IEEE Intern. Conf. Pattern Recognit."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Bleyer, M., Rother, C., and Kohli, P. (2010, January 13\u201318). Surface Stereo with Soft Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA.","DOI":"10.1109\/CVPR.2010.5539783"},{"key":"ref_30","first-page":"902","article-title":"Asymmetrical occlusion handling using graph cut for multi-view stereo","volume":"2","author":"Wei","year":"2005","journal-title":"IEEE Intern. Conf. Pattern Recognit."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2121","DOI":"10.1109\/TPAMI.2012.46","article-title":"A quantitative evaluation of confidence measures for stereo vision","volume":"34","author":"Hu","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_32","unstructured":"Liu, Z., Han, Z., Ye, Q., and Jiao, J. (2009, January 9\u201312). A New Segment-Based Algorithm for Stereo Matching. Proceedings of the IEEE International Conference on Mechatronics and Automation, Changchun, China."},{"key":"ref_33","unstructured":"Rao, A., Srihari, R.K., and Zhang, Z. (1999, January 9\u201311). Spatial color histograms for content-based image retrieval. Proceedings of the 11th IEEE International Conference on Tools with Artificial Intelligence, Chicago, IL, USA."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lempitsky, V., Rother, C., and Blake, A. (2007, January 14\u201321). Logcut-Efficient Graph Cut Optimization for Markov Random Fields. Proceedings of the IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.","DOI":"10.1109\/ICCV.2007.4408907"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1222","DOI":"10.1109\/34.969114","article-title":"Fast approximate energy minimization via graph cuts","volume":"23","author":"Boykov","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1109\/TPAMI.2004.1262177","article-title":"What energy functions can be minimized via graph cuts?","volume":"26","author":"Kolmogorov","year":"2004","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1274","DOI":"10.1109\/TPAMI.2007.1031","article-title":"Minimizing nonsubmodular functions with graph cuts-a review","volume":"29","author":"Kolmogorov","year":"2007","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_38","unstructured":"Boros, E., Hammer, P.L., and Tavares, G. (2006). Preprocessing of Unconstrained Quadratic Binary Optimization, Rutgers University. Technical Report RRR 10-2006, RUTCOR Research Report."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lempitsky, V., Roth, S., and Rother, C. (2008, January 23\u201328). FusionFlow: Discrete-Continuous Optimization for Optical Flow Estimation. Proceedings of the Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587751"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ishikawa, H. (2014, January 23\u201328). Higher-Order Clique Reduction Without Auxiliary Variables. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.177"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1109\/TPAMI.2008.99","article-title":"Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion handling","volume":"31","author":"Yang","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1094","DOI":"10.1109\/TPAMI.2012.167","article-title":"Joint depth map and color consistency estimation for stereo images with different illuminations and cameras","volume":"35","author":"Heo","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_43","first-page":"300","article-title":"Weighted Joint Bilateral Filter with Slope Depth Compensation Filter for Depth Map Refinement","volume":"2","author":"Matsuo","year":"2013","journal-title":"VISAPP"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Rhemann, C., Hosni, A., Bleyer, M., Rother, C., and Gelautz, M. (2011, January 20\u201325). Fast Cost-volume Filtering for Visual Correspondence and Beyond. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2011.5995372"},{"key":"ref_45","first-page":"195","article-title":"High-accuracy stereo depth maps using structured light","volume":"1","author":"Scharstein","year":"2013","journal-title":"IEEE Comput. Vis. Pattern Recognit."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Chakrabarti, A., Xiong, Y., Gortler, S.J., and Zickler, T. (2014). Low-level vision by consensus in a spatial hierarchy of regions.","DOI":"10.1109\/CVPR.2015.7299027"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.imavis.2015.01.003","article-title":"Robust stereo matching using adaptive random walk with restart algorithm","volume":"37","author":"Lee","year":"2015","journal-title":"Image Vis. Comput."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Spangenberg, R., Langner, T., Adfeldt, S., and Rojas, R. (2014, January 8\u201311). Large Scale Semi-Global Matching on the CPU. Proceedings of the IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, MI, USA.","DOI":"10.1109\/IVS.2014.6856419"},{"key":"ref_49","unstructured":"Middlebury Benchmark. Available online: vision.middlebury.edu\/stereo\/."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Hirschmuller, H., and Scharstein, D. (2007, January 17\u201322). Evaluation of Cost Functions for Stereo Matching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383248"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Scharstein, D., and Pal, C. (2007, January 17\u201322). Learning Conditional Random Fields for Stereo. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383191"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Yang, Q., Yang, R., Davis, J., and Nist\u00e9r, D. (2007, January 17\u201322). Spatial-depth super resolution for range images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383211"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"5559","DOI":"10.1109\/TIP.2014.2361034","article-title":"High-Quality Depth Map Upsampling and Completion for RGB-D Cameras","volume":"23","author":"Park","year":"2014","journal-title":"IEEE Trans Image Process."},{"key":"ref_54","first-page":"291","article-title":"An application of markov random fields to range sensing","volume":"5","author":"Diebel","year":"2005","journal-title":"NIPS"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Park, J., Kim, H., Tai, Y.W., Brown, M.S., and Kweon, I. (2011, January 6\u201313). High Quality Depth Map Upsampling for 3D-TOF Cameras. Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126423"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1336","DOI":"10.1016\/j.cviu.2009.11.004","article-title":"Fusion of range and color images for denoising and resolution enhancement with a non-local filter","volume":"114","author":"Huhle","year":"2010","journal-title":"Comput. Vis. image Underst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/8\/20894\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:47:12Z","timestamp":1760215632000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/15\/8\/20894"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,8,21]]},"references-count":56,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2015,8]]}},"alternative-id":["s150820894"],"URL":"https:\/\/doi.org\/10.3390\/s150820894","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2015,8,21]]}}}