{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T12:27:24Z","timestamp":1764937644159,"version":"3.41.2"},"reference-count":32,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T00:00:00Z","timestamp":1701043200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Navigating robots with precision in complex environments remains a significant challenge. In this article, we present an innovative approach to enhance robot localization in dynamic and intricate spaces like homes and offices. We leverage Visual Question Answering (VQA) techniques to integrate semantic insights into traditional mapping methods, formulating a novel position hypothesis generation to assist localization methods, while also addressing challenges related to mapping accuracy and localization reliability. Our methodology combines a probabilistic approach with the latest advances in Monte Carlo Localization methods and Visual Language models. The integration of our hypothesis generation mechanism results in more robust robot localization compared to existing approaches. Experimental validation demonstrates the effectiveness of our approach, surpassing state-of-the-art multi-hypothesis algorithms in both position estimation and particle quality. This highlights the potential for accurate self-localization, even in symmetric environments with large corridor spaces. Furthermore, our approach exhibits a high recovery rate from deliberate position alterations, showcasing its robustness. By merging visual sensing, semantic mapping, and advanced localization techniques, we open new horizons for robot navigation. Our work bridges the gap between visual perception, semantic understanding, and traditional mapping, enabling robots to interact with their environment through questions and enrich their map with valuable insights. The code for this project is available on GitHub <jats:ext-link>https:\/\/github.com\/juandpenan\/topology_nav_ros2<\/jats:ext-link>.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1290584","type":"journal-article","created":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T06:17:42Z","timestamp":1701065862000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["A visual questioning answering approach to enhance robot localization in indoor environments"],"prefix":"10.3389","volume":"17","author":[{"given":"Juan Diego","family":"Pe\u00f1a-Narvaez","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francisco","family":"Mart\u00edn","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jos\u00e9 Miguel","family":"Guerrero","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rodrigo","family":"P\u00e9rez-Rodr\u00edguez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,11,27]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2204.01691","article-title":"Do as I can, not as I say: grounding language in robotic affordances","author":"Ahn","year":"2022","journal-title":"arXiv preprint arXiv:2204.01691"},{"key":"B2","doi-asserted-by":"publisher","first-page":"132564","DOI":"10.1109\/ACCESS.2022.3230590","article-title":"OG-SGG: ontology-guided scene graph generation\u2014a case study in transfer learning for telepresence robotics","volume":"10","author":"Amodeo","year":"2022","journal-title":"IEEE Access"},{"key":"B3","first-page":"213","article-title":"\u201cEnd-to-end object detection with transformers,\u201d","volume-title":"European Conference on Computer Vision","author":"Carion","year":"2020"},{"key":"B4","first-page":"963","article-title":"\u201cActing under uncertainty: discrete Bayesian models for mobile-robot navigation,\u201d","volume-title":"Proceedings of IEEE\/RSJ International Conference on Intelligent Robots and Systems, IROS '96","author":"Cassandra","year":"1996"},{"key":"B5","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1109\/SMC.2018.00221","article-title":"\u201cRobust 2D indoor localization through laser slam and visual slam fusion,\u201d","volume-title":"2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)","author":"Chan","year":"2018"},{"key":"B6","doi-asserted-by":"publisher","first-page":"2068","DOI":"10.3390\/s20072068","article-title":"A review of visual-lidar fusion based simultaneous localization and mapping","volume":"20","author":"Debeunne","year":"2020","journal-title":"Sensors"},{"key":"B7","doi-asserted-by":"crossref","DOI":"10.15607\/RSS.2021.XVII.044","article-title":"\u201cMQA: answering the question via robotic manipulation,\u201d","volume-title":"Robotics: Science and Systems XVII","author":"Deng","year":"2021"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.03378","article-title":"PaLM-E: an embodied multimodal language model","author":"Driess","year":"2023","journal-title":"arXiv preprint arXiv:2303.03378"},{"key":"B9","doi-asserted-by":"crossref","first-page":"2555","DOI":"10.1109\/ROBOT.1992.220057","article-title":"\u201cError correction in mobile robot map learning,\u201d","author":"Engelson","year":"1992","journal-title":"Proceedings 1992 IEEE International Conference on Robotics and Automation"},{"key":"B10","doi-asserted-by":"crossref","first-page":"1933","DOI":"10.1109\/ICRA48891.2023.10160957","article-title":"\u201cPortable multi-hypothesis Monte Carlo localization for mobile robots,\u201d","volume-title":"2023 IEEE International Conference on Robotics and Automation (ICRA)","author":"Garcia","year":"2023"},{"key":"B11","doi-asserted-by":"publisher","first-page":"169","DOI":"10.3390\/machines10030169","article-title":"Text-MCL: autonomous mobile robot localization in similar environment using text-level semantic information","volume":"10","author":"Ge","year":"2022","journal-title":"Machines"},{"key":"B12","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1007\/s10514-012-9321-0","article-title":"OctoMap: an efficient probabilistic 3d mapping framework based on octrees","volume":"34","author":"Hornung","year":"2013","journal-title":"Auton. Robots"},{"key":"B13","doi-asserted-by":"crossref","first-page":"10608","DOI":"10.1109\/ICRA48891.2023.10160969","article-title":"\u201cVisual language maps for robot navigation,\u201d","volume-title":"2023 IEEE International Conference on Robotics and Automation (ICRA)","author":"Huang","year":"2023"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2207.05608","article-title":"Inner monologue: embodied reasoning through planning with language models","author":"Huang","year":"2022","journal-title":"arXiv preprint arXiv:2207.05608"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1612.06890","article-title":"CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning","author":"Johnson","year":"2016","journal-title":"arXiv preprint arXiv:1612.06890"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2104.12763","article-title":"MDETR\u2013modulated detection for end-to-end multi-modal understanding","author":"Kamath","year":"2021","journal-title":"arXiv preprint arXiv:2104.12763"},{"key":"B17","doi-asserted-by":"publisher","first-page":"6074","DOI":"10.1126\/scirobotics.abm6074","article-title":"Robot operating system 2: design, architecture, and uses in the wild","volume":"7","author":"Macenski","year":"2022","journal-title":"Sci. Robot."},{"key":"B18","doi-asserted-by":"crossref","DOI":"10.1109\/IROS45743.2020.9341207","article-title":"\u201cThe marathon 2: a navigation system,\u201d","volume-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Macenski","year":"2020"},{"key":"B19","doi-asserted-by":"publisher","first-page":"104493","DOI":"10.1016\/j.robot.2023.104493","article-title":"From the desks of ROS maintainers: a survey of modern and capable mobile robotics algorithms in the robot operating system 2","volume":"168","author":"Macenski","year":"2023","journal-title":"Robot. Auton. Syst."},{"key":"B20","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1007\/978-3-540-74024-7_44","article-title":"\u201cA comparative analysis of particle filter based localization methods,\u201d","volume-title":"RoboCup 2006: Robot Soccer World Cup X","author":"Marchetti","year":"2007"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1089","DOI":"10.1109\/TII.2015.2462771","article-title":"Improving reliability of particle filter-based localization in wireless sensor networks via hybrid particle\/FIR filtering","volume":"11","author":"Pak","year":"2015","journal-title":"IEEE Trans. Indus. Inform."},{"key":"B22","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1007\/11681120_15","article-title":"\u201cRobust Monte-Carlo localization using adaptive likelihood models,\u201d","volume-title":"European Robotics Symposium 2006","author":"Pfaff","year":"2006"},{"key":"B23","first-page":"1352","article-title":"\u201cSLAM++: simultaneous localisation and mapping at the level of objects,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Salas-Moreno","year":"2013"},{"key":"B24","doi-asserted-by":"publisher","first-page":"568","DOI":"10.3929\/ethz-b-000019956","article-title":"Map-matching of GPS traces on high-resolution navigation networks using the multiple hypothesis technique (MHT)","author":"Schuessler","year":"2009","journal-title":"Arbeitsberichte Verkehrs-und Raumplanung"},{"key":"B25","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1016\/j.isatra.2009.09.009","article-title":"Using a LRF sensor in the Kalman-filtering-based localization of a mobile robot","volume":"49","author":"Tesli\u0107","year":"2010","journal-title":"ISA Trans."},{"key":"B26","article-title":"Tidybot: personalized robot assistance with large language models","author":"Wu","year":"2023","journal-title":"arXiv preprint arXiv:2305.05658"},{"key":"B27","doi-asserted-by":"crossref","first-page":"5245","DOI":"10.1109\/ICRA48506.2021.9561471","article-title":"\u201cMSTSL: multi-sensor based two-step localization in geometrically symmetric environments,\u201d","volume-title":"2021 IEEE International Conference on Robotics and Automation (ICRA)","author":"Wu","year":"2021"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.11736","article-title":"Robotic skill acquisition via instruction augmentation with vision-language models","author":"Xiao","year":"2023","journal-title":"arXiv preprint arXiv:2211.11736"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.1109\/TIM.2022.3180425","article-title":"Multihypothesis Gaussian belief propagation for radio ranging-based localization and mapping","author":"Xiong","year":"2022","journal-title":"IEEE Trans. Instrument. Meas"},{"key":"B30","doi-asserted-by":"crossref","first-page":"4739","DOI":"10.23919\/ChiCC.2019.8866200","article-title":"\u201cSlam algorithm analysis of mobile robot based on lidar,\u201d","volume-title":"2019 Chinese Control Conference (CCC)","author":"Xuexi","year":"2019"},{"key":"B31","doi-asserted-by":"crossref","first-page":"3526","DOI":"10.1109\/ROBOT.2007.364018","article-title":"\u201cMulti-hypothesis outdoor localization using multiple visual features with a rough map,\u201d","volume-title":"Proceedings 2007 IEEE International Conference on Robotics and Automation","author":"Yun","year":"2007"},{"key":"B32","doi-asserted-by":"publisher","first-page":"596","DOI":"10.3390\/s16050596","article-title":"Smartphone-based indoor localization with bluetooth low energy beacons","volume":"16","author":"Zhuang","year":"2016","journal-title":"Sensors"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1290584\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T06:18:01Z","timestamp":1701065881000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1290584\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,27]]},"references-count":32,"alternative-id":["10.3389\/fnbot.2023.1290584"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1290584","relation":{},"ISSN":["1662-5218"],"issn-type":[{"type":"electronic","value":"1662-5218"}],"subject":[],"published":{"date-parts":[[2023,11,27]]},"article-number":"1290584"}}