{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T11:09:01Z","timestamp":1785409741990,"version":"3.56.0"},"reference-count":28,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T00:00:00Z","timestamp":1778544000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science and Technology Council","award":["114-2221-E-239-024"],"award-info":[{"award-number":["114-2221-E-239-024"]}]},{"award":["114-2221-E-239-024"],"award-info":[{"award-number":["114-2221-E-239-024"]}],"id":[{"id":"https:\/\/ror.org\/02kv4zf79","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Visually impaired users face significant challenges in navigating complex indoor environments due to limited spatial awareness and lack of real-time semantic guidance. This paper proposes a multimodal navigation system integrating environmental perception with vision-language models (VLMs). It provides context-aware and explainable guidance without requiring additional infrastructure. The proposed system combines RTAB-Map for localization, YOLO-World for open-vocabulary object detection, and a lightweight language model for semantic reasoning and natural language interaction. To evaluate our system, experiments are conducted using the RePOPE benchmark to assess hallucination in vision-language understanding. Real-world indoor navigation experiments are also performed. The results show that integrating perception with language-based reasoning improves precision by up to 2.29% and consistently enhances F1-score compared to baseline VLM approaches. Real-world experiments further demonstrate reliable navigation performance, including multi-floor path planning and obstacle-aware guidance. Hence, the proposed system effectively enhances spatial understanding and reduces hallucination, providing a practical and scalable solution for assistive navigation.<\/jats:p>","DOI":"10.3390\/s26103045","type":"journal-article","created":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T12:33:37Z","timestamp":1778589217000},"page":"3045","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Multimodal Navigation System for Visually Impaired Users Using Environmental Perception and Vision-Language Models"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6476-6625","authenticated-orcid":false,"given":"Huei-Yung","family":"Lin","sequence":"first","affiliation":[{"name":"Department of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu-Hsiang","family":"Fan","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chin-Chen","family":"Chang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Engineering, National United University, Miaoli 360302, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,5,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1109\/TCE.2018.2812498","article-title":"Virtual-blind-road following-based wearable navigation device for blind people","volume":"64","author":"Bai","year":"2018","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lee, K., Shrivastava, A., and Kacorri, H. (2020, January 1\u20135). Hand-priming in object localization for assistive egocentric vision. Proceedings of the 2020 IEEE\/CVF Winter Conference on Applications of Computer Vision, Snowmass, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093353"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tseng, Y.Y., Bell, A., and Gurari, D. (2022, January 23\u201327). VizWiz-FewShot: Locating objects in images taken by people with visual impairments. Proceedings of the 17th European Conference on Computer Vision (ECCV 2022), Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-20074-8_33"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Tsirmpas, C., Rompas, A., Anastasiou, A., and Koutsouris, D. (2014, January 3\u20135). Radio frequency identification (RFID) enhanced indoor navigation framework for seniors. Proceedings of the 2014 4th International Conference on Wireless Mobile Communication and Healthcare-Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH), Athens, Greece.","DOI":"10.4108\/icst.mobihealth.2014.257368"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Yamashita, A., Sato, K., Sato, S., and Matsubayashi, K. (2016, January 25\u201328). Development of a pedestrian navigation system for the visually impaired with QZSS and RFID. Proceedings of the 2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS), Sapporo, Japan.","DOI":"10.1109\/SCIS-ISIS.2016.0131"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mahida, P.T., Shahrestani, S., and Cheung, H. (2019, January 2\u20134). Indoor positioning framework for visually impaired people using internet of things. Proceedings of the 2019 13th International Conference on Sensing Technology (ICST), Sydney, Australia.","DOI":"10.1109\/ICST46873.2019.9047704"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kolakowski, M. (2017, January 20\u201322). Kalman filter based localization in hybrid BLE-UWB positioning system. Proceedings of the 2017 IEEE International Conference on RFID Technology & Application (RFID-TA), Warsaw, Poland.","DOI":"10.1109\/RFID-TA.2017.8098889"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Rui, C., Liu, Y., Shen, J., Li, Z., and Xie, Z. (2021, January 17\u201319). A multi-sensory blind guidance system based on YOLO and ORB-SLAM. Proceedings of the 2021 IEEE International Conference on Progress in Informatics and Computing (PIC), Shanghai, China.","DOI":"10.1109\/PIC53636.2021.9687018"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ou, W., Zhang, J., Peng, K., Yang, K., Jaworek, G., M\u00fcller, K., and Stiefelhagen, R. (2022, January 11\u201315). Indoor navigation assistance for visually impaired people via dynamic slam and panoptic segmentation with an RGB-D sensor. Proceedings of the International Conference on Computers Helping People with Special Needs, Lecco, Italy.","DOI":"10.1007\/978-3-031-08648-9_19"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"130165","DOI":"10.1109\/ACCESS.2024.3454571","article-title":"SLAM for visually impaired people: A survey","volume":"12","author":"Bamdad","year":"2024","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"702","DOI":"10.1109\/TMC.2018.2842751","article-title":"Vision-based mobile indoor assistive navigation aid for blind people","volume":"18","author":"Li","year":"2018","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Feng, D., Li, C., Zhang, Y., Yu, C., and Schwertfeger, S. (2023, January 4\u20139). OSMAG: Hierarchical semantic topometric area graph maps in the OSM format for mobile robotics. Proceedings of the 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO), Koh Samui, Thailand.","DOI":"10.1109\/ROBIO58561.2023.10354976"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2531513","DOI":"10.1109\/TIM.2023.3326167","article-title":"A topological semantic mapping method based on text-based unsupervised image segmentation for assistive indoor navigation","volume":"72","author":"Sun","year":"2023","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Xie, F., and Schwertfeger, S. (2024). Empowering robot path planning with large language models: OSMAG map topology & hierarchy comprehension with LLMs. arXiv.","DOI":"10.1109\/ROBIO64047.2024.10907658"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1002\/rob.21831","article-title":"RTAB-Map as an open-source LiDAR and visual simultaneous localization and mapping library for large-scale and long-term online operation","volume":"36","author":"Michaud","year":"2019","journal-title":"J. Field Robot."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Cheng, T., Song, L., Ge, Y., Liu, W., Wang, X., and Shan, Y. (2024, January 16\u201322). YOLO-World: Real-time open-vocabulary object detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.01599"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1109\/TRO.2015.2463671","article-title":"ORB-SLAM: A versatile and accurate monocular slam system","volume":"31","author":"Montiel","year":"2015","journal-title":"IEEE Trans. Robot."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chen, Z., Liu, Z., Wang, K., Wang, K., and Lian, S. (2024, January 14\u201318). A large vision language model based environment perception system for visually impaired people. Proceedings of the 2024 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/IROS58592.2024.10801813"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hao, Y., Magay, A., Huang, H., Yuan, S., Wen, C., and Fang, Y. (2024, January 14\u201318). ChatMap: A wearable platform based on the multi-modal foundation model to augment spatial cognition for people with blindness and low vision. Proceedings of the 2024 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/IROS58592.2024.10801606"},{"key":"ref_20","unstructured":"Wang, H., Qin, J., Bastola, A., Chen, X., Suchanek, J., Gong, Z., and Razi, A. (2024). VisionGPT: LLM-assisted real-time anomaly detection for safe visual navigation. arXiv."},{"key":"ref_21","unstructured":"Zhao, Y., Zhang, Y., Xiang, R., Li, J., and Li, H. (2024). VIALM: A survey and benchmark of visually impaired assistance with large models. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Fan, Y.H., and Lin, H.Y. (2026, January 3\u20135). A multimodal visually impaired navigation interface based on environmental perception and visual language model. Proceedings of the IEEE International Conference on Consumer Electronics (ICCE 2026), Dubai, United Arab Emirates.","DOI":"10.1109\/ICCE67443.2026.11449885"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Xu, J., Wang, C., Li, Y., Huang, X., Zhao, M., Shen, Z., Liu, Y., Wan, Y., Sun, F., and Zhang, J. (2025). Multimodal navigation and virtual companion system: A wearable device assisting blind people in independent travel. Sensors, 25.","DOI":"10.3390\/s25134223"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, H., Falletta, N.J., Xie, J., Yu, R., Lee, S., Billah, S.M., and Carroll, J.M. (2025, January 12\u201315). Enhancing the travel experience for people with visual impairments through multimodal interaction: NaviGPT, A real-time AI-driven mobile navigation system. Proceedings of the ACM International Conference Supporting Group Work (GROUP 2025), Hilton Head Island, SC, USA.","DOI":"10.1145\/3688828.3699636"},{"key":"ref_25","unstructured":"Joo, M.H., Jafari, M.K., and Taheri, A. (2026). PISHYAR: A socially intelligent smart cane for indoor social navigation and multimodal human-robot interaction for visually impaired people. arXiv."},{"key":"ref_26","unstructured":"Neuhaus, Y., and Hein, M. (2025). RePOPE: Impact of annotation errors on the POPE benchmark. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., and Wen, J.R. (2023). Evaluating object hallucination in large vision-language models. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.20"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S.J., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common objects in context. Proceedings of the 13th European Conference on Computer Vision (ECCV 2014), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/10\/3045\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T04:24:25Z","timestamp":1779337465000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/10\/3045"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,12]]},"references-count":28,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,5]]}},"alternative-id":["s26103045"],"URL":"https:\/\/doi.org\/10.3390\/s26103045","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,12]]}}}