{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:00:10Z","timestamp":1760230810182,"version":"build-2065373602"},"reference-count":44,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T00:00:00Z","timestamp":1660608000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the National Natural Science Foundation of China","award":["42071340"],"award-info":[{"award-number":["42071340"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Visual localization is a core part of many computer vision and geospatial perception applications; however, the ever-changing time phase and environment present challenges. Moreover, the ever-enriching spatial data types and sensors create new conditions for visual localization. Based on the prior 3D model and the location sensor, the current study proposes a visual localization method using semantic information. This method integrates panoptic segmentation and the matching network to refine the sensor\u2019s position and orientation and complete the target perception. First, the panoptic segmentation and the match network are used together to segment and match the 3D- model-rendered image and the truth image. The matching results are then optimized based on the semantic results. Second, the semantic consistency score is introduced in the RANSAC process to estimate the optimal 6 degree-of-freedom (6DOF) pose. In the final stage, the estimated 6DOF pose, the instance segmentation results, and the depth information are used to locate the target. Experimental results show that the proposed method is a significant improvement on advanced methods for the long-term visual localization benchmark dataset. Additionally, the proposed method is seen to provide improved localization accuracy and is capable of accurately perceiving the target for self-collected data.<\/jats:p>","DOI":"10.3390\/rs14163983","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T03:15:27Z","timestamp":1660706127000},"page":"3983","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Visual Localization and Target Perception Based on Panoptic Segmentation"],"prefix":"10.3390","volume":"14","author":[{"given":"Kefeng","family":"Lv","sequence":"first","affiliation":[{"name":"School of Surveying and Mapping, Information Engineering University, Zhengzhou 450001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongsheng","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Surveying and Mapping, Information Engineering University, Zhengzhou 450001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ying","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Surveying and Mapping, Information Engineering University, Zhengzhou 450001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenchao","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Surveying and Mapping, Information Engineering University, Zhengzhou 450001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lei","family":"Li","sequence":"additional","affiliation":[{"name":"School of Surveying and Mapping, Information Engineering University, Zhengzhou 450001, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"key":"ref_1","first-page":"1137","article-title":"Progress and challenges of geospatial artificial intelligence","volume":"50","author":"Zhang","year":"2021","journal-title":"Acta Geod. Et Cartogr. Sin."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2190","DOI":"10.1109\/TVCG.2022.3150520","article-title":"Duplicated Reality for Co-located Augmented Reality Collaboration","volume":"28","author":"Yu","year":"2022","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"25010","DOI":"10.1109\/JSEN.2020.3034139","article-title":"Multi-Sensor Fusion Tracking Algorithm Based on Augmented Reality System","volume":"21","author":"Wang","year":"2021","journal-title":"IEEE Sens. J."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Herb, M., Lemberger, M., Schmitt, M.M., Kurz, A., Weiherer, T., Navab, N., and Tombari, F. (October, January 27). Semantic Image Alignment for Vehicle Localization. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic.","DOI":"10.1109\/IROS51168.2021.9636517"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Guo, C., Lin, M., Guo, H., Liang, P., and Cheng, E. (October, January 27). Coarse-to-fine Semantic Localization with HD Map for Autonomous Driving in Structural Scenes. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic.","DOI":"10.1109\/IROS51168.2021.9635923"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"K\u00e4lin, U., Staffa, L., Grimm, D.E., and Wendt, A. (2022). Highly Accurate Pose Estimation as a Reference for Autonomous Vehicles in Near-Range Scenarios. Remote Sens., 14.","DOI":"10.20944\/preprints202112.0206.v1"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Chen, Z., Jacobson, A., S\u00fcnderhauf, N., Upcroft, B., Liu, L., Shen, C., Reid, I.D., and Milford, M. (June, January 29). Deep learning features at scale for visual place recognition. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.","DOI":"10.1109\/ICRA.2017.7989366"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"S\u00fcnderhauf, N., Shirazi, S., Jacobson, A., Dayoub, F., Pepperell, E., Upcroft, B., and Milford, M. (2015, January 13\u201317). Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free. Proceedings of the Robotics: Science and Systems Conference, Rome, Italy.","DOI":"10.15607\/RSS.2015.XI.022"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"814","DOI":"10.1109\/TPAMI.2019.2941876","article-title":"Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization?","volume":"43","author":"Sattler","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.1109\/TPAMI.2017.2711011","article-title":"NetVLAD: CNN Architecture for Weakly Supervised Place Recognition","volume":"40","author":"Gronat","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Wang, R., Wan, W., Di, K., Chen, R., and Feng, X. (2019). A High-Accuracy Indoor-Positioning Method with Automated RGB-D Image Database Construction. Remote Sens., 11.","DOI":"10.3390\/rs11212572"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wilson, D., Alshaabi, T., Van Oort, C., Zhang, X., Nelson, J., and Wshah, S. (2022). Object Tracking and Geo-Localization from Street Images. Remote Sens., 14.","DOI":"10.3390\/rs14112575"},{"key":"ref_13","first-page":"1455","article-title":"City-Scale Localization for Cameras with Known Vertical Direction","volume":"39","author":"Enqvist","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, L., Li, H., and Dai, Y. (2017, January 22\u201329). Efficient Global 2D-3D Matching for Camera Localization in a Large-Scale 3D Map. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.260"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1744","DOI":"10.1109\/TPAMI.2016.2611662","article-title":"Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization","volume":"39","author":"Sattler","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Germain, H., Bourmaud, G., and Lepetit, V. (2019, January 16\u201319). Sparse-to-Dense Hypercolumn Matching for Long-Term Visual Localization. Proceedings of the 2019 International Conference on 3D Vision (3DV), Quebec City, QC, Canada.","DOI":"10.1109\/3DV.2019.00063"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Shi, T., Shen, S., Gao, X., and Zhu, L. (2019, January 22\u201325). Visual Localization Using Sparse Semantic 3D Map. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8802957"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.1109\/LRA.2021.3136867","article-title":"Keeping an Eye on Things: Deep Learned Features for Long-Term Visual Localization","volume":"7","author":"Gridseth","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Spencer, J., Bowden, R., and Hadfield, S. (2020, January 13\u201319). Same Features, Different Day: Weakly Supervised Feature Learning for Seasonal Invariance. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00649"},{"key":"ref_20","unstructured":"Georges, B., Olivier, S., Kevin, K., and Marc, P. (2012). Large Scale Visual Geo-Localization of Images in Mountainous Terrain. The European Conference on Computer Vision, Springer."},{"key":"ref_21","first-page":"1693","article-title":"Evaluation on Geo-registration Accuracy of Outdoor Augmented Reality","volume":"32","author":"Deng","year":"2020","journal-title":"J. Syst. Simul."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"He, X., Pan, S., Gao, W., and Lu, X. (2022). LiDAR-Inertial-GNSS Fusion Positioning System in Urban Environment: Local Accurate Registration and Global Drift-Free. Remote Sens., 14.","DOI":"10.3390\/rs14092104"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Sun, J., Shen, Z., Wang, Y., Bao, H., and Zhou, X. (2021, January 20\u201325). LoFTR: Detector-Free Local Feature Matching with Transformers. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00881"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"DeTone, D., Malisiewicz, T., and Rabinovich, A. (2018, January 18\u201322). SuperPoint: Self-Supervised Interest Point Detection and Description. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPRW.2018.00060"},{"key":"ref_25","unstructured":"Carl, T., Erik, S., Lars, H., Lucas, B., Marc, P., Torsten, S., and Fredrik, K. (2018). Semantic match consistency for long-term visual localization. European Conference on Computer Vision, Springer."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Stenborg, E., Toft, C., and Hammarstrand, L. (2018, January 21\u201325). Long-Term Visual Localization Using Semantically Segmented Images. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.","DOI":"10.1109\/ICRA.2018.8463150"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., and Sivic, J. (2018, January 18\u201323). Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00897"},{"key":"ref_28","first-page":"17864","article-title":"Per-Pixel Classification is Not All You Need for Semantic Segmentation","volume":"34","author":"Cheng","year":"2021","journal-title":"Adv. Neural Inf. Processing Syst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Sarlin, P.-E., Detone, D., Malisiewicz, T., and Rabinovich, A. (2020, January 13\u201319). SuperGlue: Learning Feature Matching with Graph Neural Networks. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00499"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2074","DOI":"10.1109\/TPAMI.2020.3032010","article-title":"Long-Term Visual Localization Revisited","volume":"44","author":"Toft","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Dusmanu, M., Rocco, I., Pajdla, T., Pollefeys, M., Sivic, J., Torii, A., and Sattler, T. (2019, January 15\u201320). D2-Net: A Trainable CNN for Joint Description and Detection of Local Features. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00828"},{"key":"ref_32","first-page":"12405","article-title":"R2d2: Reliable and repeatable detector and descriptor","volume":"32","author":"Revaud","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1109\/JAS.2021.1003907","article-title":"Domain-Invariant Similarity Activation Map Contrastive Learning for Retrieval-Based Long-Term Visual Localization","volume":"9","author":"Hu","year":"2022","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_34","unstructured":"Shi, T., Cui, H., Song, Z., and Shen, S. (2020). Dense Semantic 3D Map Based Long-Term Visual Localization with Hybrid Features. arXiv."},{"key":"ref_35","unstructured":"Fan, H., Zhou, Y., Li, A., Gao, S., Li, J., and Guo, Y. (2020). Visual Localization Using Semantic Segmentation and Depth Prediction. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Larsson, M., Stenborg, E., Toft, C., Hammarstrand, L., Sattler, T., and Kahl, F. (November, January 27). Fine-Grained Segmentation Networks: Self-Supervised Segmentation for Improved Long-Term Visual Localization. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.","DOI":"10.1109\/ICCV.2019.00012"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020). End-to-End Object Detection with Transformers, Springer.","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Zhai, R., and Yuan, Y. (2022). A Method of Vision Aided GNSS Positioning Using Semantic Information in Complex Urban Environment. Remote Sens., 14.","DOI":"10.3390\/rs14040869"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kirillov, A., He, K., Girshick, R., Rother, C., and Girshick, P. (2019, January 15\u201320). Panoptic Segmentation. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00963"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Cheng, B., Collins, M.D., Zhu, Y., Liu, T., Huang, T.S., Adam, H., and Chen, L.C. (2020, January 13\u201319). Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01249"},{"key":"ref_41","first-page":"3145407","article-title":"Instance and Panoptic Segmentation Using Conditional Convolutions","volume":"2022","author":"Tian","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_42","unstructured":"Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Diaz-Zapata, M., Erkent, \u00d6., and Laugier, C. (2021, January 12\u201316). YOLO-based Panoptic Segmentation Network. Proceedings of the 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), Madrid, Spain.","DOI":"10.1109\/COMPSAC51774.2021.00170"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1109\/TRO.2017.2705103","article-title":"ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras","volume":"33","year":"2017","journal-title":"IEEE Trans. Robot."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/16\/3983\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:10:32Z","timestamp":1760141432000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/14\/16\/3983"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,16]]},"references-count":44,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["rs14163983"],"URL":"https:\/\/doi.org\/10.3390\/rs14163983","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2022,8,16]]}}}