{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T07:42:10Z","timestamp":1767339730122,"version":"build-2065373602"},"reference-count":44,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2020,7,3]],"date-time":"2020-07-03T00:00:00Z","timestamp":1593734400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>We propose a completely unsupervised approach to simultaneously estimate scene depth, ego-pose, ground segmentation and ground normal vector from only monocular RGB video sequences. In our approach, estimation for different scene structures can mutually benefit each other by the joint optimization. Specifically, we use the mutual information loss to pre-train the ground segmentation network and before adding the corresponding self-learning label obtained by a geometric method. By using the static nature of the ground and its normal vector, the scene depth and ego-motion can be efficiently learned by the self-supervised learning procedure. Extensive experimental results on both Cityscapes and KITTI benchmark demonstrate the significant improvement on the estimation accuracy for both scene depth and ego-pose by our approach. We also achieve an average error of about 3     \u2218     for estimated ground normal vectors. By deploying our proposed geometric constraints, the IOU accuracy of unsupervised ground segmentation is increased by 35% on the Cityscapes dataset.<\/jats:p>","DOI":"10.3390\/s20133737","type":"journal-article","created":{"date-parts":[[2020,7,6]],"date-time":"2020-07-06T09:49:11Z","timestamp":1594028951000},"page":"3737","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Joint Unsupervised Learning of Depth, Pose, Ground Normal Vector and Ground Segmentation by a Monocular Camera Sensor"],"prefix":"10.3390","volume":"20","author":[{"given":"Lu","family":"Xiong","sequence":"first","affiliation":[{"name":"Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongkun","family":"Wen","sequence":"additional","affiliation":[{"name":"Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuyao","family":"Huang","sequence":"additional","affiliation":[{"name":"Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7864-3255","authenticated-orcid":false,"given":"Junqiao","family":"Zhao","sequence":"additional","affiliation":[{"name":"Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5085-7219","authenticated-orcid":false,"given":"Wei","family":"Tian","sequence":"additional","affiliation":[{"name":"Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University, Shanghai 201804, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,7,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Khan, S.M., and Shah, M. (2006). A multiview approach to tracking people in crowded scenes using a planar homography constraint. European Conference on Computer Vision, Springer.","DOI":"10.1007\/11744085_11"},{"key":"ref_2","unstructured":"Chen, X., Kundu, K., Zhu, Y., Berneshawi, A.G., Ma, H., Fidler, S., and Urtasun, R. (2015). 3d object proposals for accurate object class detection. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ku, J., Mozifian, M., Lee, J., Harakeh, A., and Waslander, S.L. (2018, January 1\u20135). Joint 3d proposal generation and object detection from view aggregation. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8594049"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"274","DOI":"10.1016\/j.cviu.2005.04.005","article-title":"Extraction, matching, and pose recovery based on dominant rectangular structures","volume":"100","author":"Zhang","year":"2005","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Forss\u00e9n, P.E., and Lowe, D.G. (2007, January 14\u201320). Shape descriptors for maximally stable extremal regions. Proceedings of the 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil.","DOI":"10.1109\/ICCV.2007.4409025"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bian, J., Lin, W.Y., Matsushita, Y., Yeung, S.K., Nguyen, T.D., and Cheng, M.M. (2017, January 21\u201326). Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.302"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hartley, R., and Zisserman, A. (2003). Multiple View Geometry in Computer Vision, Cambridge University Press.","DOI":"10.1017\/CBO9780511811685"},{"key":"ref_8","unstructured":"Bian, J.W., Wu, Y.H., Zhao, J., Liu, Y., Zhang, L., Cheng, M.M., and Reid, I. (2019). An evaluation of feature matchers for fundamental matrix estimation. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017, January 21\u201326). Unsupervised learning of depth and ego-motion from video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.700"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mahjourian, R., Wicke, M., and Angelova, A. (2018, January 18\u201322). Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00594"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yin, Z., and Shi, J. (2018, January 18\u201322). Geonet: Unsupervised learning of dense depth, optical flow and camera pose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00212"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zou, Y., Luo, Z., and Huang, J.B. (2018, January 8\u201314). Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01228-1_3"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ranjan, A., Jampani, V., Balles, L., Kim, K., Sun, D., Wulff, J., and Black, M.J. (2019, January 16\u201320). Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01252"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"McDaniel, M.W., Nishihata, T., Brooks, C.A., and Iagnemma, K. (2010, January 3\u20138). Ground plane identification using LIDAR in forested environments. Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA.","DOI":"10.1109\/ROBOT.2010.5509963"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Man, Y., Weng, X., Li, X., and Kitani, K. (2018). GroundNet: Monocular Ground Plane Estimation with Geometric Consistency. ACM International Conference on Multimedia, Association for Computing Machinery.","DOI":"10.1145\/3343031.3351068"},{"key":"ref_16","unstructured":"Bansal, A., Russell, B., and Gupta, A. (July, January June). Marr revisited: 2d-3d alignment via surface normal prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Eigen, D., and Fergus, R. (2015, January 7\u201313). Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.304"},{"key":"ref_18","unstructured":"Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (1, January June). The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision meets robotics: The kitti dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Rob. Res."},{"key":"ref_20","unstructured":"Eigen, D., Puhrsch, C., and Fergus, R. (2014). Depth map prediction from a single image using a multi-scale deep network. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_21","first-page":"1228","article-title":"Refinenet: Multi-path refinement networks for dense prediction","volume":"42","author":"Lin","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kuznietsov, Y., Stuckler, J., and Leibe, B. (2017, January 21\u201326). Semi-supervised deep learning for monocular depth map prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.238"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yin, Z., Darrell, T., and Yu, F. (2019, January 16\u201320). Hierarchical discrete distribution decomposition for match density estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00620"},{"key":"ref_24","unstructured":"Tang, C., and Tan, P. (2018). Ba-net: Dense bundle adjustment network. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Godard, C., Mac Aodha, O., and Brostow, G.J. (2017, January 21\u201326). Unsupervised monocular depth estimation with left-right consistency. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.699"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhan, H., Garg, R., Saroj Weerasekera, C., Li, K., Agarwal, H., and Reid, I. (2018, January 18\u201323). Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00043"},{"key":"ref_27","unstructured":"Bian, J., Li, Z., Wang, N., Zhan, H., Shen, C., Cheng, M.M., and Reid, I. (2019). Unsupervised scale-consistent depth and ego-motion learning from monocular video. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_28","unstructured":"Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M. (2017). Learning discrete representations via information maximizing self-augmented training. arXiv."},{"key":"ref_29","unstructured":"Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. (2018). Learning deep representations by mutual information estimation and maximization. arXiv."},{"key":"ref_30","unstructured":"Yosinski, J., Clune, J., Nguyen, A., Fuchs, T.J., and Lipson, H. (2015). Understanding Neural Networks through Deep Visualization. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ji, X., Henriques, J.F., and Vedaldi, A. (2019, January 16\u201320). Invariant information clustering for unsupervised image classification and segmentation. Proceedings of the IEEE International Conference on Computer Vision, Long Beach, CA, USA.","DOI":"10.1109\/ICCV.2019.00996"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1007\/s11263-006-0031-y","article-title":"Recovering surface layout from an image","volume":"75","author":"Hoiem","year":"2007","journal-title":"Int. J. Comput. Vis."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ren, Z., and Jae Lee, Y. (2018, January 18\u201323). Cross-domain self-supervised multi-task feature learning using synthetic imagery. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00086"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, W., Xiang, D., and Deng, J. (2017, January 21\u201326). Surface normals in the wild. Proceedings of the IEEE International Conference on Computer Vision, Honolulu, HI, USA.","DOI":"10.1109\/ICCV.2017.173"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Song, S., Tan, P., and Xiao, J. (2014). Panocontext: A whole-room 3d context model for panoramic scene understanding. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-10599-4_43"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Stekovic, S., Fraundorfer, F., and Lepetit, V. (2020). General 3D Room Layout from a Single View by Render-and-Compare. arXiv.","DOI":"10.1007\/978-3-030-58517-4_12"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Yang, Z., Wang, P., Xu, W., Zhao, L., and Nevatia, R. (2017). Unsupervised learning of geometry with edge-aware depth-normal consistency. arXiv.","DOI":"10.1609\/aaai.v32i1.12257"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Wang, C., Miguel Buenaposada, J., Zhu, R., and Lucey, S. (2018, January 18\u201323). Learning depth from monocular videos using direct methods. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00216"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Qiu, J., Cui, Z., Zhang, Y., Zhang, X., Liu, S., Zeng, B., and Pollefeys, M. (2019, January 16\u201320). DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00343"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2024","DOI":"10.1109\/TPAMI.2015.2505283","article-title":"Learning depth from single monocular images using deep convolutional neural fields","volume":"38","author":"Liu","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Garg, R., BG, V.K., Carneiro, G., and Reid, I. (2016). Unsupervised cnn for single view depth estimation: Geometry to the rescue. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-319-46484-8_45"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1109\/TRO.2015.2463671","article-title":"ORB-SLAM: A versatile and accurate monocular SLAM system","volume":"31","author":"Montiel","year":"2015","journal-title":"IEEE Trans. Rob."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Dragon, R., and Van Gool, L. (2014, January 23\u201328). Ground plane estimation using a hidden markov model. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.442"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/13\/3737\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:47:10Z","timestamp":1760176030000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/13\/3737"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,3]]},"references-count":44,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2020,7]]}},"alternative-id":["s20133737"],"URL":"https:\/\/doi.org\/10.3390\/s20133737","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,7,3]]}}}