{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T16:02:46Z","timestamp":1781193766568,"version":"3.54.1"},"reference-count":57,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2018,2,1]],"date-time":"2018-02-01T00:00:00Z","timestamp":1517443200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Geometric surface information such as depth maps and surface normals can be acquired by various methods such as stereo light fields, shape from shading and photometric stereo techniques. We compare several algorithms which deal with the combination of depth with surface normal information in order to reconstruct a refined depth map. The reasons for performance differences are examined from the perspective of alternative formulations of surface normals for depth reconstruction. We review and analyze methods in a systematic way. Based on our findings, we introduce a new generalized fusion method, which is formulated as a least squares problem and outperforms previous methods in the depth error domain by introducing a novel normal weighting that performs closer to the geodesic distance measure. Furthermore, a novel method is introduced based on Total Generalized Variation (TGV) which further outperforms previous approaches in terms of the geodesic normal distance error and maintains comparable quality in the depth error domain.<\/jats:p>","DOI":"10.3390\/s18020431","type":"journal-article","created":{"date-parts":[[2018,2,2]],"date-time":"2018-02-02T04:20:50Z","timestamp":1517545250000},"page":"431","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["A Review of Depth and Normal Fusion Algorithms"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2083-0135","authenticated-orcid":false,"given":"Doris","family":"Antensteiner","sequence":"first","affiliation":[{"name":"Center for Vision, Automation and Control, Austrian Institute of Technology, Vienna 1210, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Svorad","family":"\u0160tolc","sequence":"additional","affiliation":[{"name":"Center for Vision, Automation and Control, Austrian Institute of Technology, Vienna 1210, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Pock","sequence":"additional","affiliation":[{"name":"Center for Vision, Automation and Control, Austrian Institute of Technology, Vienna 1210, Austria"},{"name":"Institute of Computer Graphics and Vision, Graz University of Technology, Graz 8010, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2018,2,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1023\/A:1014554110407","article-title":"Real-Time Correlation-Based Stereo Vision with Reduced Border Errors","volume":"47","author":"Innocent","year":"2002","journal-title":"Int. J. Comput. Vis."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1109\/TPAMI.2007.1166","article-title":"Stereo Processing by Semiglobal Matching and Mutual Information","volume":"30","author":"Hirschmuller","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Haller, I., Pantilie, C., Oniga, F., and Nedevschi, S. (2010, January 21\u201324). Real-Time Semi-Global Dense Stereo Solution with Improved Sub-Pixel Accuracy. Proceedings of the 2010 IEEE Intelligent Vehicles Symposium, San Diego, CA, USA.","DOI":"10.1109\/IVS.2010.5548104"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"779","DOI":"10.1016\/j.image.2013.04.002","article-title":"Optimized Hierarchical Block Matching for Fast and Accurate Image Registration","volume":"28","author":"Je","year":"2013","journal-title":"Signal Process. Image Commun."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Yang, X., Chen, X., and Xi, J. (2017, January 22\u201325). Block Based Dense Stereo Matching Using Adaptive Cost Aggregation and Limited Disparity Estimation. Proceedings of the 2017 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Turin, Italy.","DOI":"10.1109\/I2MTC.2017.7969972"},{"key":"ref_6","unstructured":"Chen, X., Li, D., and Zou, J. (2017, January 2\u20134). Depth Estimation of Stereo Matching Based on Microarray Camera. Proceedings of the 2017 2nd International Conference on Image, Vision and Computing (ICIVC), Chengdu, China."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bolles, R.C., Baker, H.H., and Marimont, D.H. (1987). Epipolar-Plane Image Analysis: An Approach to Determining Structure from Motion, Springer.","DOI":"10.1007\/BF00128525"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wanner, S., Straehle, C., and Goldluecke, B. (2013, January 23\u201328). Globally Consistent Multi-Label Assignment on the Ray Space of 4D Light Fields. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.135"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"73:1","DOI":"10.1145\/2461912.2461926","article-title":"Scene Reconstruction from High Spatio-Angular Resolution Light Fields","volume":"32","author":"Kim","year":"2013","journal-title":"ACM Trans. Graph."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Tosic, I., and Berkner, K. (2014, January 23\u201328). Light Field Scale-Depth Space Transform for Dense Depth Estimation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Columbus, OH, USA.","DOI":"10.1109\/CVPRW.2014.71"},{"key":"ref_11","unstructured":"Tao, M.W., Wang, T.C., Malik, J., and Ramamoorthi, R. (2016, January 8\u201316). Depth Estimation for Glossy Surfaces with Light-Field Cameras. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands."},{"key":"ref_12","unstructured":"Longuet-Higgins, H.C. (1987). Readings in Computer Vision: Issues, Problems, Principles, and Paradigms, Morgan Kaufmann Publishers Inc."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Snavely, N., Seitz, S.M., and Szeliski, R. (2006). Photo Tourism: Exploring Photo Collections in 3D, ACM.","DOI":"10.1145\/1141911.1141964"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Park, J., Kim, H., Tai, Y.W., Brown, M.S., and Kweon, I. (2011, January 6\u201313). High Quality Depth Map Upsampling for 3D-ToF Cameras. Proceedings of the 2011 IEEE International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126423"},{"key":"ref_15","first-page":"191139","article-title":"Photometric Method for Determining Surface Orientation from Multiple Images","volume":"19","author":"Woodham","year":"1980","journal-title":"Int. Soc. Opt. Photonics"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1109\/34.3909","article-title":"A Method for Enforcing Integrability in Shape From Shading Algorithms","volume":"10","author":"Frankot","year":"1988","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","unstructured":"Wu, T.P., and Tang, C.K. (2005, January 20\u201325). Dense Photometric Stereo Using a Mirror Sphere and Graph Cut. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Diego, CA, USA."},{"key":"ref_18","unstructured":"Malis, E., and Vargas, M. (2007). Deeper Understanding of the Homography Decomposition for Vision-Based Control, INRIA. Research Report RR-6303."},{"key":"ref_19","unstructured":"K\u00f6ser, K. (2008). Geometric Estimation with Local Affine Frames and Free-form Surfaces. [Ph.D. Thesis, Christian-Albrechts-Universit\u00e4t zu Kiel]."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Braz, J., Pettr\u00e9, J., Richard, P., Kerren, A., Linsen, L., Battiato, S., and Imai, F. (2016). Novel Methods for Estimating Surface Normals from Affine Transformations. Communications in Computer and Information Science, Springer International Publishing.","DOI":"10.1007\/978-3-319-29971-6"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Eigen, D., and Fergus, R. (2015, January 7\u201313). Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.304"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wu, C., Wilburn, B., Matsushita, Y., and Theobalt, C. (2011, January 20\u201325). High-Quality Shape from Multi-View Stereo and Shading Under General Illumination. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.","DOI":"10.1109\/CVPR.2011.5995388"},{"key":"ref_23","unstructured":"Zhang, L., Curless, B., Hertzmann, A., and Seitz, S.M. (2003, January 13\u201316). Shape and Motion under Varying Illumination: Unifying Structure from Motion, Photometric Stereo, and Multi-view Stereo. Proceedings of the 9th IEEE International Conference on Computer Vision, Nice, France."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Joshi, N., and Kriegman, D.J. (2007, January 14\u201321). Shape from Varying Illumination and Viewpoint. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.","DOI":"10.1109\/ICCV.2007.4409021"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"548","DOI":"10.1109\/TPAMI.2007.70820","article-title":"Multiview Photometric Stereo","volume":"30","author":"Esteban","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1145\/1073204.1073226","article-title":"Efficiently Combining Positions and Normals for Precise 3D Geometry","volume":"24","author":"Nehab","year":"2005","journal-title":"ACM Trans. Graph."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Haque, S.M., Chatterjee, A., and Govindu, V.M. (2014, January 23\u201328). High Quality Photometric Reconstruction Using a Depth Camera. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.292"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1591","DOI":"10.1109\/TPAMI.2016.2608944","article-title":"Robust Multiview Photometric Stereo Using Planar Mesh Parameterization","volume":"39","author":"Park","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Han, Y., Lee, J.Y., and Kweon, I.S. (2013, January 1\u20138). High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, NSW, Australia.","DOI":"10.1109\/ICCV.2013.204"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Chatterjee, A., and Govindu, V.M. (2015, January 7\u201312). Photometric refinement of depth maps for multi-albedo objects. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298695"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Shi, B., Inose, K., Matsushita, Y., Tan, P., Yeung, S.K., and Ikeuchi, K. (2014, January 8\u201311). Photometric Stereo Using Internet Images. Proceedings of the 2014 2nd International Conference on 3D Vision, Tokyo, Japan.","DOI":"10.1109\/3DV.2014.9"},{"key":"ref_32","unstructured":"Yu, H. (2012, January 16\u201321). Edge-preserving Photometric Stereo via Depth Fusion. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR \u201912, Providence, RI, USA."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ti, C., Yang, R., Davis, J., and Pan, Z. (2015, January 7\u201312). Simultaneous Time-of-Flight sensing and photometric stereo with a single ToF sensor. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299062"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yu, L.F., Yeung, S.K., Tai, Y.W., and Lin, S. (2013, January 23\u201328). Shading-Based Shape Refinement of RGB-D Images. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, CVPR\u201913, Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.186"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wu, C., Zollh\u00f6fer, M., Nie\u00dfner, M., Stamminger, M., Izadi, S., and Theobalt, C. (2014). Real-Time Shading-Based Refinement for Consumer Depth Cameras. ACM Trans. Graph., 33.","DOI":"10.1145\/2661229.2661232"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Kadambi, A., Taamazyan, V., Shi, B., and Raskar, R. (2015, January 7\u201313). Polarized 3D: High-Quality Depth Sensing with Polarization Cues. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.385"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Antensteiner, D., \u0160tolc, S., Valentin, K., Blaschitz, B., Huber-M\u00f6rk, R., and Pock, T. (February, January 29). High-Precision 3D Sensing with Hybrid Light Field and Photometric Stereo Approach in Multi-Line Scan Framework. Proceedings of the IS&T International Symposium on Electronic Imaging: Intelligent Robotics and Industrial Applications using Computer Vision 2017, Burlingame, CA, USA.","DOI":"10.2352\/ISSN.2470-1173.2017.9.IRIACV-268"},{"key":"ref_38","unstructured":"Horn, B.K.P., and Brooks, M.J. (1989). Shape from Shading, MIT Press."},{"key":"ref_39","first-page":"19","article-title":"Depth and All-In-Focus Imaging by a Multi-Line-Scan Light-Field Camera","volume":"23","author":"Soukup","year":"2014","journal-title":"J. Electron. Imaging"},{"key":"ref_40","unstructured":"Kazhdan, M., Bolitho, M., and Hoppe, H. (2006, January 26\u201328). Poisson Surface Reconstruction. Proceedings of the Fourth Eurographics Symposium on Geometry Processing, SGP\u201906, Cagliari, Sardinia."},{"key":"ref_41","unstructured":"Heber, S. (2015). Variational Shape from Unconventional Imaging Devices. [Ph.D. Thesis, Graz University of Technology]."},{"key":"ref_42","unstructured":"Scherr, T. (2017). Gradient-Based Surface Reconstruction and the Application to Wind Waves. [Master\u2019s Thesis, Ruprecht-Karls-University Heidelberg]."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Rostami, M., Michailovich, O., and Wang, Z. (October, January 30). Gradient-Based Surface Reconstruction Using Compressed Sensing. Proceedings of the 2012 19th IEEE International Conference on Image Processing, Orlando, FL, USA.","DOI":"10.1109\/ICIP.2012.6467009"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Harker, M., and O\u2019Leary, P. (2008, January 23\u201328). Least Squares Surface Reconstruction from Measured Gradient Fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587414"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Du, Z., Robles-Kelly, A., and Lu, F. (2007, January 3\u20135). Robust Surface Reconstruction from Gradient Field Using the L1 Norm. Proceedings of the 9th Biennial Conference of the Australian Pattern Recognition Society on Digital Image Computing Techniques and Applications (DICTA 2007), Glenelg, Australia.","DOI":"10.1109\/DICTA.2007.4426797"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Agrawal, A., Chellappa, R., and Raskar, R. (2005, January 17\u201321). An Algebraic Approach to Surface Reconstruction from Gradient Fields. Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV\u201905), Beijing, China.","DOI":"10.1109\/ICCV.2005.31"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/0734-189X(86)90114-3","article-title":"The Variational Approach to Shape from Shading","volume":"33","author":"Horn","year":"1986","journal-title":"Comput. Vis. Graph. Image Process."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Nesterov, Y.E. (2003). Introductory Lectures on Convex Optimization: A Basic Course, Kluwer Academic Publishers.","DOI":"10.1007\/978-1-4419-8853-9"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1137\/080716542","article-title":"A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems","volume":"2","author":"Beck","year":"2009","journal-title":"SIAM J. Imaging Sci."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1137\/090769521","article-title":"Total Generalized Variation","volume":"3","author":"Bredies","year":"2010","journal-title":"SIAM J. Imaging Sci."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Chambolle, A., and Pock, T. (2016). An Introduction to Continuous Optimization for Imaging, Cambridge University Press.","DOI":"10.1017\/S096249291600009X"},{"key":"ref_52","unstructured":"(2017, November 10). The Stanford 3D Scanning Repository. Available online: http:\/\/graphics.stanford.edu\/data\/3Dscanrep\/."},{"key":"ref_53","unstructured":"(2017, November 10). Persistence of Vision Pty. Ltd. (2004) [Computer Software], Available online: http:\/\/www.povray.org\/download\/."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Parikh, N., and Boyd, S. (2013). Proximal Algorithms, Now Publishers Inc.","DOI":"10.1561\/9781601987174"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Chambolle, A., and Pock, T. (2011). A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging, Springer.","DOI":"10.1007\/s10851-010-0251-1"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1007\/s10107-004-0552-5","article-title":"Smooth Minimization of Non-Smooth Functions","volume":"103","author":"Nesterov","year":"2005","journal-title":"Math. Program."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Beck, A., and Teboulle, M. (2010). Gradient-Based Algorithms with Applications to Signal Recovery Problems, Cambridge University Press.","DOI":"10.1017\/CBO9780511804458.003"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/2\/431\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:53:28Z","timestamp":1760194408000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/2\/431"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,1]]},"references-count":57,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2018,2]]}},"alternative-id":["s18020431"],"URL":"https:\/\/doi.org\/10.3390\/s18020431","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,2,1]]}}}