{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T17:50:23Z","timestamp":1779126623213,"version":"3.51.4"},"reference-count":53,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2021,8,13]],"date-time":"2021-08-13T00:00:00Z","timestamp":1628812800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61673039"],"award-info":[{"award-number":["61673039"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Self-Programming Intelligent Robot Project of Ji Hua Laboratory","award":["No. X190101TB190"],"award-info":[{"award-number":["No. X190101TB190"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Monocular depth estimation based on unsupervised learning has attracted great attention due to the rising demand for lightweight monocular vision sensors. Inspired by multi-task learning, semantic information has been used to improve the monocular depth estimation models. However, multi-task learning is still limited by multi-type annotations. As far as we know, there are scarcely any large public datasets that provide all the necessary information. Therefore, we propose a novel network architecture Semantic-Feature-Aided Monocular Depth Estimation Network (SFA-MDEN) to extract multi-resolution depth features and semantic features, which are merged and fed into the decoder, with the goal of predicting depth with the support of semantics. Instead of using loss functions to relate the semantics and depth, the fusion of feature maps for semantics and depth is employed to predict the monocular depth. Therefore, two accessible datasets with similar topics for depth estimation and semantic segmentation can meet the requirements of SFA-MDEN for training sets. We explored the performance of the proposed SFA-MDEN with experiments on different datasets, including KITTI, Make3D, and our own dataset BHDE-v1. The experimental results demonstrate that SFA-MDEN achieves competitive accuracy and generalization capacity compared to state-of-the-art methods.<\/jats:p>","DOI":"10.3390\/s21165476","type":"journal-article","created":{"date-parts":[[2021,8,15]],"date-time":"2021-08-15T22:51:27Z","timestamp":1629067887000},"page":"5476","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["SFA-MDEN: Semantic-Feature-Aided Monocular Depth Estimation Network Using Dual Branches"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4813-8514","authenticated-orcid":false,"given":"Rui","family":"Wang","sequence":"first","affiliation":[{"name":"Digital Photoelectric Information Processing Technology Laboratory, School of Instrumentation and Optoelectronic Engineering, Beihang University, Beijing 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jialing","family":"Zou","sequence":"additional","affiliation":[{"name":"Digital Photoelectric Information Processing Technology Laboratory, School of Instrumentation and Optoelectronic Engineering, Beihang University, Beijing 100191, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James Zhiqing","family":"Wen","sequence":"additional","affiliation":[{"name":"Engineering Research Center for Intelligent Robotics, Ji Hua Laboratory, Foshan 528200, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Alidoost, F., Arefi, H., and Tombari, F. (2019). 2D image-to-3D model: Knowledge-based 3D building reconstruction (3DBR) using single aerial images and convolutional neural networks (CNNs). Remote Sens., 11.","DOI":"10.3390\/rs11192219"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1016\/j.image.2018.09.013","article-title":"Object instance detection with pruned alexnet and extended data","volume":"70","author":"Wang","year":"2019","journal-title":"Signal Process Image Commun."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Stathopoulou, E.K., Battisti, R., Cernea, D., Remondino, F., and Georgopoulos, A. (2021). Semantically derived geometric constraints for MVS reconstruction of textureless areas. Remote Sens., 13.","DOI":"10.3390\/rs13061053"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Jin, L., Wang, X., He, M., and Wang, J. (2021). DRNet: A depth-based regression network for 6D object pose estimation. Sensors, 21.","DOI":"10.3390\/s21051692"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hwang, S.-J., Park, S.-J., Kim, G.-M., and Baek, J.-H. (2021). Unsupervised monocular depth estimation for colonoscope system using feedback network. Sensors, 21.","DOI":"10.3390\/s21082691"},{"key":"ref_6","unstructured":"Microsoft (2021, April 23). Kinect for Windows. Available online: https:\/\/developer.microsoft.com\/zh-cn\/windows\/kinect\/."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1489","DOI":"10.1109\/21.44067","article-title":"Structure from stereo\u2014A review","volume":"19","author":"Dhond","year":"1989","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Khan, F., Salahuddin, S., and Javidnia, H. (2020). Deep learning-based monocular depth estimation methods\u2014A state-of-the-art review. Sensors, 20.","DOI":"10.3390\/s20082272"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16\u201321). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2012.6248074"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1109\/TPAMI.2008.132","article-title":"Make3D: Learning 3D scene structure from a single still image","volume":"31","author":"Saxena","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1007\/s11263-007-0071-y","article-title":"3-D depth reconstruction from a single still image","volume":"76","author":"Saxena","year":"2008","journal-title":"Int. J. Comput. Vis."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2024","DOI":"10.1109\/TPAMI.2015.2505283","article-title":"Learning depth from single monocular images using deep convolutional neural fields","volume":"38","author":"Liu","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2144","DOI":"10.1109\/TPAMI.2014.2316835","article-title":"Depth transfer: Depth extraction from video using non-parametric sampling","volume":"36","author":"Karsch","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","unstructured":"Eigen, D., Puhrsch, C., and Fergus, R. (2014, January 8\u201313). Depth map prediction from a single image using a multi-scale deep network. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Eigen, D., and Fergus, R. (2015, January 7\u201313). Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.304"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., and Navab, N. (2016, January 25\u201328). Deeper depth prediction with fully convolutional residual networks. Proceedings of the 2016 Fourth International Conference on 3D Vision, Stanford, CA, USA.","DOI":"10.1109\/3DV.2016.32"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wofk, D., Ma, F., Yang, T.-J., Karaman, S., and Sze, V. (2019, January 20\u201324). Fast monocular depth estimation on embedded systems. Proceedings of the 2019 International Conference on Robotics and Automation, Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794182"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"89300","DOI":"10.1109\/ACCESS.2020.2993494","article-title":"Learning depth for scene reconstruction using an encoder-decoder model","volume":"8","author":"Tu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_20","unstructured":"Lee, J.H., Han, M.K., Ko, D.W., and Suh, I.H. (2020). From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Fu, H., Gong, M., Wang, C., Batmanghelich, K., and Tao, D. (2018, January 18\u201323). Deep ordinal regression network for monocular depth estimation. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00214"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"114930","DOI":"10.1109\/ACCESS.2020.3003466","article-title":"Soft regression of monocular depth using scale-semantic exchange network","volume":"8","author":"Su","year":"2020","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kim, S., Nam, J., and Ko, B. (2019). Fast depth estimation in a single image using lightweight efficient neural network. Sensors, 19.","DOI":"10.3390\/s19204434"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kuznietsov, Y., St\u00fcckler, J., and Leibe, B. (2017, January 21\u201326). Semi-supervised deep learning for monocular depth map prediction. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.238"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Atapour-Abarghouei, A., and Breckon, T.P. (2018, January 18\u201323). Real-time monocular depth estimation using synthetic data with domain adaptation via image style transfer. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00296"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhao, S., Fu, H., Gong, M., and Tao, D. (2019, January 15\u201320). Geometry-aware symmetric domain adaptation for monocular depth estimation. Proceedings of 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01002"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2410","DOI":"10.1109\/TPAMI.2019.2936024","article-title":"Semi-supervised adversarial monocular depth estimation","volume":"42","author":"Ji","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Garg, R., Bg, V.K., Carneiro, G., and Reid, I. (2016, January 11\u201314). Unsupervised CNN for single view depth estimation: Geometry to the rescue. Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46484-8_45"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Godard, C., Mac Aodha, O., and Brostow, G.J. (2017, January 21\u201326). Unsupervised monocular depth estimation with left-right consistency. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.699"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Goldman, M., Hassner, T., and Avidan, S. (2019, January 16\u201317). Learn stereo, infer mono: Siamese networks for self-supervised, monocular, depth estimation. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00348"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Poggi, M., Aleotti, F., Tosi, F., and Mattoccia, S. (2018, January 1\u20135). Towards real-time unsupervised monocular depth estimation on CPU. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593814"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ye, X., Zhang, M., Xu, R., Zhong, W., Fan, X., Liu, Z., and Zhang, J. (2019, January 8\u201312). Unsupervised Monocular depth estimation based on dual attention mechanism and depth-aware loss. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China.","DOI":"10.1109\/ICME.2019.00037"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zhang, C., Liu, J., and Han, C. (2020, January 13\u201314). Unsupervised learning of depth estimation based on attention model from monocular images. Proceedings of the 2020 International Conference on Virtual Reality and Visualization (ICVRV), Recife, Brazil.","DOI":"10.1109\/ICVRV51359.2020.00046"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ling, C., Zhang, X., and Chen, H. (2021). Unsupervised monocular depth estimation using attention and multi-warp reconstruction. IEEE Trans. Multimed.","DOI":"10.1109\/TMM.2021.3091308"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"155612","DOI":"10.1109\/ACCESS.2019.2947084","article-title":"Facilitating PTZ camera auto-calibration to be noise resilient with two images","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017, January 21\u201326). Unsupervised learning of depth and ego-motion from video. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.700"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Mahjourian, R., Wicke, M., and Angelova, A. (2018, January 18\u201323). Unsupervised learning of depth and ego-motion from monocular video using 3D geometric constraints. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00594"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1145\/3386569.3392377","article-title":"Consistent video depth estimation","volume":"39","author":"Luo","year":"2020","journal-title":"ACM Trans. Graph."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Godard, C., Mac Aodha, O., Firman, M., and Brostow, G. (November, January 27). Digging into self-supervised monocular depth estimation. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00393"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ramamonjisoa, M., Du, Y., and Lepetit, V. (2020, January 13\u201319). Predicting sharp and accurate occlusion boundaries in monocular depth estimation using displacement fields. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01466"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Casser, V., Pirk, S., Mahjourian, R., and Angelova, A. (2018). Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos. arXiv.","DOI":"10.1609\/aaai.v33i01.33018001"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2548","DOI":"10.1007\/s11263-021-01484-6","article-title":"Unsupervised scale-consistent depth learning from video","volume":"129","author":"Bian","year":"2021","journal-title":"Int. J. Comput. Vis."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1016\/j.isprsjprs.2020.06.004","article-title":"MiniNet: An extremely lightweight convolutional neural network for real-time unsupervised monocular depth estimation","volume":"166","author":"Liu","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Wang, C., Buenaposada, J.M., Zhu, R., and Lucey, S. (2018, January 18\u201323). Learning depth from monocular videos using direct methods. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00216"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Yin, Z., and Shi, J. (2018, January 18\u201323). GeoNet: Unsupervised learning of dense depth, optical flow and camera pose. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00212"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Cipolla, R., Gal, Y., and Kendall, A. (2018, January 18\u201323). Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00781"},{"key":"ref_47","unstructured":"Ramirez, P.Z., Poggi, M., Tosi, F., Mattoccia, S., and Di Stefano, L. (2018, January 2\u20136). Geometry meets semantics for semi-supervised monocular depth estimation. Proceedings of the 2018 Asian Conference on Computer Vision, Perth, Australia."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Mousavian, A., Pirsiavash, H., and Ko\u0161eck\u00e1, J. (2016, January 25\u201328). Joint semantic segmentation and depth estimation with deep convolutional networks. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.","DOI":"10.1109\/3DV.2016.69"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Nekrasov, V., Dharmasiri, T., Spek, A., Drummond, T., Shen, C., and Reid, I. (2019, January 20\u201324). Real-time joint semantic segmentation and depth estimation using asymmetric annotations. Proceedings of the 2019 International Conference on Robotics and Automation, Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794220"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1007\/s10846-020-01205-0","article-title":"Semi-supervised monocular depth estimation based on semantic supervision","volume":"5","author":"Yue","year":"2020","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27\u201330). The cityscapes dataset for semantic urban scene understanding. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.350"},{"key":"ref_52","unstructured":"Nekrasov, V., Shen, C., and Reid, I. (2018). Light-weight refinenet for real-time semantic segmentation. arxiv."},{"key":"ref_53","unstructured":"(2021, April 22). PHANTOM 4 PRO\/PRO+. Available online: https:\/\/dl.djicdn.com\/downloads\/phantom_4_pro\/20200108\/Phantom_4_Pro_Pro_Plus_Series_User_Manual_CHS.pdf."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/16\/5476\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:45:49Z","timestamp":1760165149000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/16\/5476"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,13]]},"references-count":53,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["s21165476"],"URL":"https:\/\/doi.org\/10.3390\/s21165476","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,13]]}}}