{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T10:07:04Z","timestamp":1767694024145,"version":"3.48.0"},"reference-count":79,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T00:00:00Z","timestamp":1767657600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T00:00:00Z","timestamp":1767657600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["Grant NO. 62322608"],"award-info":[{"award-number":["Grant NO. 62322608"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Vis. Intell."],"published-print":{"date-parts":[[2026,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models\u00a0(LLMs) such as the GPT series exhibit impressive reasoning and in-context learning capabilities due to the substantial amount of data and computational resources involved in LLM training. Some previous studies have applied the LLMs to vision-and-language navigation\u00a0(VLN) in order to create navigation agents that are entirely LLM-based, operating within a zero-shot setting, aiming to reveal and utilize LLMs\u2019 reasoning and planning capability for VLN tasks. However, these methods employ text-based LLMs for navigation agents, generating a text description of environmental observations during navigation. Other smaller LLMs are used for image-to-text translation, resulting in a gap between image-to-text translation and environmental navigation. Moreover, the high cost of advanced LLMs such as GPT-4 also hinders the application of LLM-based navigation agents. This is particularly the case given the increasing length of the context, which includes navigation history and the numerous visual images generated during navigation. In this paper, we propose NavGemini, a navigation system based entirely on the newly developed multi-modal LLM, Gemini-Pro-Vision. Our aim is to study and utilize the visual-spatial and multi-modal capabilities of LLMs in VLN tasks, while mitigating the challenges posed by token limits when LLMs process large amounts of image-based data and historical information, and when the available multi-modal LLMs perform relatively poorly. Our proposed NavGemini, with its elaborate prompts, successfully outperforms previous methods by 5.7% in terms of success rate, even when using inferior LLMs. This demonstrates the strong ability and potential of multi-modal LLM-based agents in VLN tasks.<\/jats:p>","DOI":"10.1007\/s44267-025-00105-x","type":"journal-article","created":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T09:04:29Z","timestamp":1767690269000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["NavGemini: a multi-modal LLM agent for vision-and-language navigation"],"prefix":"10.1007","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1612-3641","authenticated-orcid":false,"given":"Ganlong","family":"Zhao","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4805-0926","authenticated-orcid":false,"given":"Guanbin","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0470-5548","authenticated-orcid":false,"given":"Yizhou","family":"Yu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,1,6]]},"reference":[{"key":"105_CR1","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., Azhar, F., et\u00a0al. (2023). LLaMA: open and efficient foundation language models. arXiv preprint. arXiv:2302.13971."},{"key":"105_CR2","unstructured":"OpenAI (2023). GPT-4 technical report. arXiv preprint. arXiv:2303.08774."},{"key":"105_CR3","unstructured":"Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et\u00a0al. (2023). PaLM 2 technical report. arXiv preprint. arXiv:2305.10403."},{"key":"105_CR4","unstructured":"OpenAI (2023). GPT-4V(ision) system card. OpenAI. Retrieved November 1, 2025, from https:\/\/cdn.openai.com\/papers\/GPTV_System_Card.pdf."},{"key":"105_CR5","first-page":"540","volume-title":"Proceedings of the 7th conference on robot learning","author":"W. Huang","year":"2023","unstructured":"Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., & Li, F.-F. (2023). VoxPoser: composable 3D value maps for robotic manipulation with language models. In J. Tan, M. Toussaint, & K. Darvish (Eds.), Proceedings of the 7th conference on robot learning (pp. 540\u2013562). Retrieved November 1, 2025, from https:\/\/proceedings.mlr.press\/v229\/huang23c.html."},{"key":"105_CR6","first-page":"25081","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"Y. Mu","year":"2023","unstructured":"Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., & Luo, P. (2023). EmbodiedGPT: vision-language pre-training via embodied chain of thought. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp.\u00a025081\u201325094). Red Hook: Curran Associates."},{"key":"105_CR7","first-page":"287","volume-title":"Proceedings of the 6th conference on robot learning","author":"B. Ichter","year":"2022","unstructured":"Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., et al. (2022). Do as I can, not as I say: grounding language in robotic affordances. In K. Liu, D. Kulic, & J. Ichnowski (Eds.), Proceedings of the 6th conference on robot learning (pp.\u00a0287\u2013318). Retrieved November 1, 2025, from https:\/\/proceedings.mlr.press\/v205\/ichter23a.html."},{"key":"105_CR8","doi-asserted-by":"publisher","DOI":"10.1007\/s44267-025-00093-y","volume":"3","author":"J. Xie","year":"2025","unstructured":"Xie, J., Chen, Z., Zhang, R., & Li, G. (2025). Large multimodal agents: a survey. Visual Intelligence, 3, 24.","journal-title":"Visual Intelligence"},{"key":"105_CR9","first-page":"3674","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"P. Anderson","year":"2018","unstructured":"Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., S\u00fcnderhauf, N., Reid, I. D., Gould, S., & van den Hengel, A. (2018). Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a03674\u20133683). Piscataway: IEEE."},{"key":"105_CR10","first-page":"104","volume-title":"Proceedings of the 16th European conference on computer vision","author":"J. Krantz","year":"2020","unstructured":"Krantz, J., Wijmans, E., Majumdar, A., Batra, D., & Lee, S. (2020). Beyond the nav-graph: vision-and-language navigation in continuous environments. In A. Vedaldi, H. Bischof, T. Brox, & J. Frahm (Eds.), Proceedings of the 16th European conference on computer vision (pp.\u00a0104\u2013120). Cham: Springer."},{"key":"105_CR11","first-page":"7641","volume-title":"Proceedings of the 38th AAAI conference on artificial intelligence","author":"G. Zhou","year":"2024","unstructured":"Zhou, G., Hong, Y., & Wu, Q. (2024). NavGPT: explicit reasoning in vision-and-language navigation with large language models. In M. J. Wooldridge, J. G. Dy, & S. Natarajan (Eds.), Proceedings of the 38th AAAI conference on artificial intelligence (pp.\u00a07641\u20137649). Palo Alto: AAAI Press."},{"key":"105_CR12","first-page":"9796","volume-title":"Proceedings of the 62nd annual meeting of the Association for Computational Linguistics","author":"J. Chen","year":"2024","unstructured":"Chen, J., Lin, B., Xu, R., Chai, Z., Liang, X., & Wong, K. K. (2024). MapGPT: map-guided prompting with adaptive path planning for vision-and-language navigation. In L. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd annual meeting of the Association for Computational Linguistics (pp.\u00a09796\u20139810). Stroudsburg: ACL."},{"key":"105_CR13","first-page":"19730","volume-title":"Proceedings of the international conference on machine learning","author":"J. Li","year":"2023","unstructured":"Li, J., Li, D., Savarese, S., & Hoi, S. C. H. (2023). BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the international conference on machine learning (pp.\u00a019730\u201319742). Retrieved December 1, 2025, from https:\/\/proceedings.mlr.press\/v202\/li23q.html."},{"key":"105_CR14","unstructured":"Gemini Team (2023). Gemini: a family of highly capable multimodal models. arXiv preprint. arXiv:2312.11805."},{"key":"105_CR15","first-page":"9979","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Y. Qi","year":"2020","unstructured":"Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W. Y., Shen, C., & van den Hengel, A. (2020). REVERIE: remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a09979\u20139988). Piscataway: IEEE."},{"key":"105_CR16","first-page":"4392","volume-title":"Proceedings of the 2020 conference on empirical methods in natural language processing","author":"A. Ku","year":"2020","unstructured":"Ku, A., Anderson, P., Patel, R., Ie, E., & Baldridge, J. (2020). Room-across-room: multilingual vision-and-language navigation with dense spatiotemporal grounding. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 conference on empirical methods in natural language processing (pp.\u00a04392\u20134412). Stroudsburg: ACL."},{"key":"105_CR17","first-page":"394","volume-title":"Proceedings of the 3rd conference on robot learning","author":"J. Thomason","year":"2019","unstructured":"Thomason, J., Murray, M., Cakmak, M., & Zettlemoyer, L. (2019). Vision-and-dialog navigation. In L. P. Kaelbling, D. Kragic, & K. Sugiura (Eds.), Proceedings of the 3rd conference on robot learning (pp.\u00a0394\u2013406). Retrieved November 1, 2025, from http:\/\/proceedings.mlr.press\/v100\/thomason20a.html."},{"key":"105_CR18","first-page":"12689","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"F. Zhu","year":"2021","unstructured":"Zhu, F., Liang, X., Zhu, Y., Yu, Q., Chang, X., & Liang, X. (2021). SOON: scenario oriented object navigation with graph-based exploration. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a012689\u201312699). Piscataway: IEEE."},{"key":"105_CR19","first-page":"1624","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"C. Liu","year":"2021","unstructured":"Liu, C., Zhu, F., Chang, X., Liang, X., Ge, Z., & Shen, Y. (2021). Vision-language navigation with random environmental mixup. In Proceedings of the IEEE\/CVF international conference on computer vision (pp.\u00a01624\u20131634). Piscataway: IEEE."},{"key":"105_CR20","first-page":"15407","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"S. Wang","year":"2022","unstructured":"Wang, S., Montgomery, C., Orbay, J., Birodkar, V., Faust, A., Gur, I., Jaques, N., Waters, A., Baldridge, J., & Anderson, P. (2022). Less is more: generating grounded navigation instructions from landmarks. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a015407\u201315417). Piscataway: IEEE."},{"key":"105_CR21","first-page":"5834","volume-title":"Proceedings of the 35th international conference on neural information processing systems","author":"S. Chen","year":"2021","unstructured":"Chen, S., Guhur, P., Schmid, C., & Laptev, I. (2021). History aware multimodal transformer for vision-and-language navigation. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, & J. W. Vaughan (Eds.), Proceedings of the 35th international conference on neural information processing systems (pp.\u00a05834\u20135847). Red Hook: Curran Associates."},{"key":"105_CR22","first-page":"8455","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"H. Wang","year":"2021","unstructured":"Wang, H., Wang, W., Liang, W., Xiong, C., & Shen, J. (2021). Structured scene memory for vision-language navigation. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a08455\u20138464). Piscataway: IEEE."},{"key":"105_CR23","first-page":"15922","volume-title":"Proceedings of the IEEE\/CVF international conference on computer vision","author":"A. Pashevich","year":"2021","unstructured":"Pashevich, A., Schmid, C., & Sun, C. (2021). Episodic transformer for vision-and-language navigation. In Proceedings of the IEEE\/CVF international conference on computer vision (pp.\u00a015922\u201315932). Piscataway: IEEE."},{"key":"105_CR24","first-page":"259","volume-title":"Proceedings of the 16th European conference on computer vision","author":"A. Majumdar","year":"2020","unstructured":"Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., & Batra, D. (2020). Improving vision-and-language navigation with image-text pairs from the web. In A. Vedaldi, H. Bischof, T. Brox, & J. Frahm (Eds.), Proceedings of the 16th European conference on computer vision (pp.\u00a0259\u2013274). Cham: Springer."},{"key":"105_CR25","first-page":"13134","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"W. Hao","year":"2020","unstructured":"Hao, W., Li, C., Li, X., Carin, L., & Gao, J. (2020). Towards learning a generic agent for vision-and-language navigation via pre-training. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a013134\u201313143). Piscataway: IEEE."},{"key":"105_CR26","first-page":"16516","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"S. Chen","year":"2022","unstructured":"Chen, S., Guhur, P., Tapaswi, M., Schmid, C., & Laptev, I. (2022). Think global, act local: dual-scale graph transformer for vision-and-language navigation. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a016516\u201316526). Piscataway: IEEE."},{"key":"105_CR27","first-page":"6710","volume-title":"Proceedings of the IEEE international conference on robotics and automation","author":"Y. Qiao","year":"2025","unstructured":"Qiao, Y., Lyu, W., Wang, H., Wang, Z., Li, Z., Zhang, Y., Tan, M., & Wu, Q. (2025). Open-Nav: exploring zero-shot vision-and-language navigation in continuous environment with open-source LLMs. In Proceedings of the IEEE international conference on robotics and automation (pp.\u00a06710\u20136717). Piscataway: IEEE."},{"key":"105_CR28","first-page":"17380","volume-title":"Proceedings of the IEEE international conference on robotics and automation","author":"Y. Long","year":"2024","unstructured":"Long, Y., Li, X., Cai, W., & Dong, H. (2024). Discuss before moving: visual language navigation via multi-expert discussions. In Proceedings of the IEEE international conference on robotics and automation (pp.\u00a017380\u201317387). Piscataway: IEEE."},{"key":"105_CR29","first-page":"5998","volume-title":"Proceedings of the 31st international conference on neural information processing systems","author":"A. Vaswani","year":"2017","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, & R. Garnett (Eds.), Proceedings of the 31st international conference on neural information processing systems (pp.\u00a05998\u20136008). Red Hook: Curran Associates."},{"key":"105_CR30","first-page":"1877","volume-title":"Proceedings of the 34th international conference on neural information processing systems","author":"T. B. Brown","year":"2020","unstructured":"Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, & H. Lin (Eds.), Proceedings of the 34th international conference on neural information processing systems (pp.\u00a01877\u20131901). Red Hook: Curran Associates."},{"key":"105_CR31","volume":"24","author":"A. Chowdhery","year":"2023","unstructured":"Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023). PaLM: scaling language modeling with pathways. Journal of Machine Learning Research, 24, 240.","journal-title":"Journal of Machine Learning Research"},{"key":"105_CR32","unstructured":"Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M. T., Li, X., Lin, X. V., et\u00a0al. (2022). OPT: open pre-trained transformer language models. arXiv preprint. arXiv:2205.01068."},{"key":"105_CR33","first-page":"1","volume-title":"Proceedings of the 10th international conference on learning representations","author":"J. Wei","year":"2022","unstructured":"Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2022). Finetuned language models are zero-shot learners. In Proceedings of the 10th international conference on learning representations (pp.\u00a01\u201346). Retrieved December 1, 2025, from https:\/\/openreview.net\/forum?id=gEZrGCozdqR."},{"key":"105_CR34","unstructured":"Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et\u00a0al. (2023). Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. Retrieved March 30, 2025, from https:\/\/lmsys.org\/blog\/2023-03-30-vicuna."},{"key":"105_CR35","first-page":"2978","volume-title":"Proceedings of the 57th conference of the Association for Computational Linguistics","author":"Z. Dai","year":"2019","unstructured":"Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-XL: attentive language models beyond a fixed-length context. In A. Korhonen, D. R. Traum, & L. M\u00e0rquez (Eds.), Proceedings of the 57th conference of the Association for Computational Linguistics (pp.\u00a02978\u20132988). Stroudsburg: ACL."},{"key":"105_CR36","first-page":"5547","volume-title":"Proceedings of the international conference on machine learning","author":"N. Du","year":"2022","unstructured":"Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. (2022). GLaM: efficient scaling of language models with mixture-of-experts. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesv\u00e1ri, G. Niu, & S. Sabato (Eds.), Proceedings of the international conference on machine learning (pp.\u00a05547\u20135569). Retrieved December 1, 2025, from https:\/\/proceedings.mlr.press\/v162\/du22c.html."},{"key":"105_CR37","volume":"23","author":"W. Fedus","year":"2022","unstructured":"Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23, 120.","journal-title":"Journal of Machine Learning Research"},{"key":"105_CR38","first-page":"30016","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"J. Hoffmann","year":"2022","unstructured":"Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp.\u00a030016\u201330030). Red Hook: Curran Associates."},{"key":"105_CR39","unstructured":"Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young, S., et\u00a0al. (2021). Scaling language models: methods, analysis & insights from training gopher. arXiv preprint. arXiv:2112.11446."},{"key":"105_CR40","volume":"25","author":"H. W. Chung","year":"2024","unstructured":"Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2024). Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25, 70.","journal-title":"Journal of Machine Learning Research"},{"key":"105_CR41","first-page":"27730","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"L. Ouyang","year":"2022","unstructured":"Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp.\u00a027730\u201327744). Red Hook: Curran Associates."},{"key":"105_CR42","unstructured":"Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023). Visual ChatGPT: talking, drawing and editing with visual foundation models. arXiv preprint. arXiv:2303.04671."},{"key":"105_CR43","first-page":"38154","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"Y. Shen","year":"2023","unstructured":"Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023). HuggingGPT: solving AI tasks with ChatGPT and its friends in hugging face. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp.\u00a038154\u201338180). Red Hook: Curran Associates."},{"key":"105_CR44","first-page":"68539","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"T. Schick","year":"2023","unstructured":"Schick, T., Dwivedi-Yu, J., Dess\u00ec, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023). Toolformer: language models can teach themselves to use tools. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp.\u00a068539\u201368551). Red Hook: Curran Associates."},{"key":"105_CR45","unstructured":"Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et\u00a0al. (2023). Check your facts and try again: improving large language models with external knowledge and automated feedback. arXiv preprint. arXiv:2302.12813."},{"key":"105_CR46","first-page":"1","volume-title":"Proceedings of the 11th international conference on learning representations","author":"S. Yao","year":"2023","unstructured":"Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., & Cao, Y. (2023). React: synergizing reasoning and acting in language models. In Proceedings of the 11th international conference on learning representations (pp.\u00a01\u201333). Retrieved December 1, 2025, from https:\/\/openreview.net\/forum?id=WE_vluYUL-X."},{"key":"105_CR47","unstructured":"Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., Shoham, Y., Bata, H., Levine, Y., Leyton-Brown, K., et\u00a0al. (2022). MRKL systems: a modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning. arXiv preprint. arXiv:2205.00445."},{"key":"105_CR48","first-page":"24824","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"J. Wei","year":"2022","unstructured":"Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp.\u00a024824\u201324837). Red Hook: Curran Associates."},{"key":"105_CR49","first-page":"23716","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"J. Alayrac","year":"2022","unstructured":"Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022). Flamingo: a visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp.\u00a023716\u201323736). Red Hook: Curran Associates."},{"key":"105_CR50","unstructured":"Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S. Y., Sagawa, S., et\u00a0al. (2023). OpenFlamingo: an open-source framework for training large autoregressive vision-language models. arXiv preprint. arXiv:2308.01390."},{"key":"105_CR51","unstructured":"Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., & Ta\u015f\u0131rlar, S. (2024). Fuyu-8B: a multimodal architecture for AI agents. Retrieved December 1, 2025, from https:\/\/www.adept.ai\/blog\/fuyu-8b."},{"key":"105_CR52","first-page":"26679","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"J. Lin","year":"2024","unstructured":"Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Han, S. (2024). VILA: on pre-training for visual language models. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a026679\u201326689). Piscataway: IEEE."},{"key":"105_CR53","unstructured":"Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et\u00a0al. (2023). Qwen technical report. arXiv preprint. arXiv:2309.16609."},{"key":"105_CR54","doi-asserted-by":"crossref","unstructured":"Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Ruiz, C. R., Goodman, S., Wang, X., Tay, Y., et\u00a0al. (2023). PaLI-X: on scaling up a multilingual vision and language model. arXiv preprint. arXiv:2305.18565.","DOI":"10.1109\/CVPR52733.2024.01368"},{"key":"105_CR55","first-page":"8469","volume-title":"Proceedings of the international conference on machine learning","author":"D. Driess","year":"2023","unstructured":"Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023). PaLM-E: an embodied multimodal language model. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the international conference on machine learning (pp.\u00a08469\u20138488). Retrieved December 1, 2025, from https:\/\/proceedings.mlr.press\/v202\/driess23a.html."},{"key":"105_CR56","first-page":"34892","volume-title":"Proceedings of the 37th international conference on neural information processing systems","author":"H. Liu","year":"2023","unstructured":"Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th international conference on neural information processing systems (pp.\u00a034892\u201334916). Red Hook: Curran Associates."},{"key":"105_CR57","unstructured":"Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et\u00a0al. (2023). mPLUG-Owl: modularization empowers large language models with multimodality. arXiv preprint. arXiv:2304.14178."},{"key":"105_CR58","first-page":"1","volume-title":"Proceedings of the 12th international conference on learning representations","author":"D. Zhu","year":"2024","unstructured":"Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2024). MiniGPT-4: enhancing vision-language understanding with advanced large language models. In Proceedings of the 12th international conference on learning representations (pp.\u00a01\u201317). Retrieved December 1, 2025, from https:\/\/openreview.net\/forum?id=1tZbq88f27."},{"key":"105_CR59","first-page":"950","volume-title":"Findings of the proceedings of the 2024 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies","author":"B. Pan","year":"2024","unstructured":"Pan, B., Panda, R., Jin, S., Feris, R., Oliva, A., Isola, P., & Kim, Y. (2024). LangNav: language as a perceptual representation for navigation. In K. Duh, H. G\u00f3mez-Adorno, & S. Bethard (Eds.), Findings of the proceedings of the 2024 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies (pp.\u00a0950\u2013974). Stroudsburg: ACL."},{"key":"105_CR60","first-page":"2165","volume-title":"Proceedings of the 7th conference on robot learning","author":"B. Zitkovich","year":"2023","unstructured":"Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. (2023). RT-2: vision-language-action models transfer web knowledge to robotic control. In J. Tan, M. Toussaint, & K. Darvish (Eds.), Proceedings of the 7th conference on robot learning (pp.\u00a02165\u20132183). Retrieved November 1, 2025, from https:\/\/proceedings.mlr.press\/v229\/zitkovich23a.html."},{"key":"105_CR61","first-page":"18924","volume-title":"Proceedings of the 38th AAAI conference on artificial intelligence","author":"R. Schumann","year":"2024","unstructured":"Schumann, R., Zhu, W., Feng, W., Fu, T., Riezler, S., & Wang, W. Y. (2024). VELMA: verbalization embodiment of LLM agents for vision and language navigation in street view. In M. J. Wooldridge, J. G. Dy, & S. Natarajan (Eds.), Proceedings of the 38th AAAI conference on artificial intelligence (pp.\u00a018924\u201318933). Palo Alto: AAAI Press."},{"key":"105_CR62","unstructured":"Hu, Y., Lin, F., Zhang, T., Yi, L., & Gao, Y. (2023). Look before you leap: unveiling the power of GPT-4V in robotic vision-language planning. arXiv preprint. arXiv:2311.17842."},{"key":"105_CR63","first-page":"9118","volume-title":"Proceedings of the international conference on machine learning","author":"W. Huang","year":"2022","unstructured":"Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022). Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesv\u00e1ri, G. Niu, & S. Sabato (Eds.), Proceedings of the international conference on machine learning (pp.\u00a09118\u20139147). Retrieved December 1, 2025, from https:\/\/proceedings.mlr.press\/v162\/huang22a.html."},{"key":"105_CR64","first-page":"11523","volume-title":"Proceedings of the IEEE international conference on robotics and automation","author":"I. Singh","year":"2023","unstructured":"Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023). ProgPrompt: generating situated robot task plans using large language models. In Proceedings of the IEEE international conference on robotics and automation (pp.\u00a011523\u201311530). Piscataway: IEEE."},{"key":"105_CR65","first-page":"3546","volume-title":"Proceedings of the IEEE\/RSJ international conference on intelligent robots and systems","author":"J. Wu","year":"2023","unstructured":"Wu, J., Antonova, R., Kan, A., Lepert, M., Zeng, A., Song, S., Bohg, J., Rusinkiewicz, S., & Funkhouser, T. A. (2023). TidyBot: personalized robot assistance with large language models. In Proceedings of the IEEE\/RSJ international conference on intelligent robots and systems (pp.\u00a03546\u20133553). Piscataway: IEEE."},{"key":"105_CR66","first-page":"1713","volume-title":"Proceedings of the 60th annual meeting of the Association for Computational Linguistics","author":"P. Sharma","year":"2022","unstructured":"Sharma, P., Torralba, A., & Andreas, J. (2022). Skill induction and planning with latent language. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), Proceedings of the 60th annual meeting of the Association for Computational Linguistics (pp.\u00a01713\u20131726). Stroudsburg: ACL."},{"key":"105_CR67","first-page":"9493","volume-title":"Proceedings of the IEEE international conference on robotics and automation","author":"J. Liang","year":"2023","unstructured":"Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023). Code as policies: language model programs for embodied control. In Proceedings of the IEEE international conference on robotics and automation (pp.\u00a09493\u20139500). Piscataway: IEEE."},{"key":"105_CR68","first-page":"25377","volume-title":"Proceedings of the 36th international conference on neural information processing systems","author":"A. C. Tam","year":"2022","unstructured":"Tam, A. C., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C. Y., Strouse, D., Wang, J., Banino, A., & Hill, F. (2022). Semantic exploration from language abstractions and pretrained representations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th international conference on neural information processing systems (pp.\u00a025377\u201325389). Red Hook: Curran Associates."},{"key":"105_CR69","first-page":"492","volume-title":"Proceedings of the 6th conference on robot learning","author":"D. Shah","year":"2022","unstructured":"Shah, D., Osinski, B., Ichter, B., & Levine, S. (2022). LM-Nav: robotic navigation with large pre-trained models of language, vision, and action. In K. Liu, D. Kulic, & J. Ichnowski (Eds.), Proceedings of the 6th conference on robot learning (pp.\u00a0492\u2013504). Retrieved November 1, 2025, from https:\/\/proceedings.mlr.press\/v205\/shah23b.html."},{"key":"105_CR70","first-page":"10608","volume-title":"Proceedings of the IEEE international conference on robotics and automation","author":"C. Huang","year":"2023","unstructured":"Huang, C., Mees, O., Zeng, A., & Burgard, W. (2023). Visual language maps for robot navigation. In Proceedings of the IEEE international conference on robotics and automation (pp.\u00a010608\u201310615). Piscataway: IEEE."},{"key":"105_CR71","first-page":"42829","volume-title":"Proceedings of the international conference on machine learning","author":"K. Zhou","year":"2023","unstructured":"Zhou, K., Zheng, K., Pryor, C., Shen, Y., Jin, H., Getoor, L., & Wang, X. E. (2023). ESC: exploration with soft commonsense constraints for zero-shot object navigation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the international conference on machine learning (pp.\u00a042829\u201342842). Retrieved December 1, 2025, from https:\/\/proceedings.mlr.press\/v202\/zhou23r.html."},{"key":"105_CR72","doi-asserted-by":"publisher","first-page":"7957","DOI":"10.18653\/v1\/2023.emnlp-main.494","volume-title":"Proceedings of the 2023 conference on empirical methods in natural language processing","author":"R. Pryzant","year":"2023","unstructured":"Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023). Automatic prompt optimization with \u201cgradient descent\u201d and beam search. In H.\u00a0Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 conference on empirical methods in natural language processing (pp.\u00a07957\u20137968). Stroudsburg: ACL."},{"key":"105_CR73","unstructured":"Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., & Chadha, A. (2024). A\u00a0systematic survey of prompt engineering in large language models: techniques and applications. arXiv preprint. arXiv:2402.07927."},{"key":"105_CR74","doi-asserted-by":"crossref","unstructured":"Ramnath, K., Zhou, K., Guan, S., Mishra, S. S., Qi, X., Shen, Z., Wang, S., Woo, S., Jeoung, S., Wang, Y., et\u00a0al. (2025). A systematic survey of automatic prompt optimization techniques. arXiv preprint. arXiv:2502.16923.","DOI":"10.18653\/v1\/2025.emnlp-main.1681"},{"key":"105_CR75","first-page":"667","volume-title":"Proceedings of the international conference on 3D vision","author":"A. X. Chang","year":"2017","unstructured":"Chang, A. X., Dai, A., Funkhouser, T. A., Halber, M., Nie\u00dfner, M., Savva, M., Song, S., Zeng, A., & Zhang, Y. (2017). Matterport3D: learning from RGB-D data in indoor environments. In Proceedings of the international conference on 3D vision (pp.\u00a0667\u2013676). Piscataway: IEEE."},{"key":"105_CR76","first-page":"3318","volume-title":"Proceedings of the 32nd international conference on neural information processing systems","author":"D. Fried","year":"2018","unstructured":"Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L., Berg-Kirkpatrick, T., Saenko, K., Klein, D., & Darrell, T. (2018). Speaker-follower models for vision-and-language navigation. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Proceedings of the 32nd international conference on neural information processing systems (pp.\u00a03318\u20133329). Red Hook: Curran Associates."},{"key":"105_CR77","first-page":"2610","volume-title":"Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies","author":"H. Tan","year":"2019","unstructured":"Tan, H., Yu, L., & Bansal, M. (2019). Learning to navigate unseen environments: back translation with environmental dropout. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies (pp.\u00a02610\u20132621). Stroudsburg: ACL."},{"key":"105_CR78","first-page":"1643","volume-title":"Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition","author":"Y. Hong","year":"2021","unstructured":"Hong, Y., Wu, Q., Qi, Y., Opazo, C. R., & Gould, S. (2021). VLN BERT: a recurrent vision-and-language BERT for navigation. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp.\u00a01643\u20131653). Piscataway: IEEE."},{"key":"105_CR79","first-page":"5099","volume-title":"Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing","author":"H. Tan","year":"2019","unstructured":"Tan, H., & Bansal, M. (2019). LXMERT: learning cross-modality encoder representations from transformers. In K. Inui, J. Jiang, V. Ng, & X. Wan (Eds.), Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (pp.\u00a05099\u20135110). Stroudsburg: ACL."}],"container-title":["Visual Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44267-025-00105-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44267-025-00105-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44267-025-00105-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T10:03:09Z","timestamp":1767693789000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44267-025-00105-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,6]]},"references-count":79,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,12]]}},"alternative-id":["105"],"URL":"https:\/\/doi.org\/10.1007\/s44267-025-00105-x","relation":{},"ISSN":["2097-3330","2731-9008"],"issn-type":[{"type":"print","value":"2097-3330"},{"type":"electronic","value":"2731-9008"}],"subject":[],"published":{"date-parts":[[2026,1,6]]},"assertion":[{"value":"25 June 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 December 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 December 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 January 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"1"}}