{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:27:13Z","timestamp":1776083233473,"version":"3.50.1"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2025,9,3]]},"abstract":"<jats:p>Understanding users' environments is crucial for determining their states, needs, and interactions with technology. This work focuses on route context, including environmental factors such as road conditions, traffic, and weather that influence users while traveling. Integrating route context with LLMs enables reasoning over environmental factors, thus allowing users to ask questions like 'When is the best moment for a phone call along my route?' or 'Is this a good route for a drive in a convertible?'. We introduce the first LLM that natively understands route context. We create ContextualRoutes1, a dataset of 320k routes, each comprising road, weather, and traffic data. We annotate these routes using a template and a teacher model to create LabeledRoutes1, a multimodal multi-task question-answering dataset with over 1k tasks and 40k conversations containing routes and text. Based on the first dataset, we train the first route context tokenizer that groups the routes into semantically meaningful clusters. On its basis, we propose the first route-context-aware LLM and find it capable of zero-shot reasoning on routes. Still, we urge that further research on learning cross-modal route-to-text understanding is necessary and discuss challenges in the future development of artifacts for this novel branch of research.<\/jats:p>","DOI":"10.1145\/3749552","type":"journal-article","created":{"date-parts":[[2025,9,3]],"date-time":"2025-09-03T17:15:45Z","timestamp":1756919745000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["RouteLLM: A Large Language Model with Native Route Context Understanding to Enable Context-Aware Reasoning"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-3522-9210","authenticated-orcid":false,"given":"Philipp","family":"Hallgarten","sequence":"first","affiliation":[{"name":"Porsche AG, Stuttgart, Germany and Technical University of Munich, Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2184-630X","authenticated-orcid":false,"given":"Verena Jasmin","family":"Hallitschke","sequence":"additional","affiliation":[{"name":"Porsche AG, Stuttgart, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3146-4484","authenticated-orcid":false,"given":"Enkelejda","family":"Kasneci","sequence":"additional","affiliation":[{"name":"Human-Centered Technologies for Learning, Technical University of Munich, Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5009-2327","authenticated-orcid":false,"given":"Michael","family":"Beigl","sequence":"additional","affiliation":[{"name":"Karlsruhe Institute of Technology, Karlsruhe, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4961-6554","authenticated-orcid":false,"given":"Tobias","family":"Grosse-Puppendahl","sequence":"additional","affiliation":[{"name":"Porsche AG, Stuttgart, Germany"}]}],"member":"320","published-online":{"date-parts":[[2025,9,3]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild. arXiv preprint arXiv:1906.02569","author":"Abid Abubakar","year":"2019","unstructured":"Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019. Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild. arXiv preprint arXiv:1906.02569 (2019)."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534573"},{"key":"e_1_2_1_3_1","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3620665.3640366"},{"key":"e_1_2_1_5_1","unstructured":"Apple Inc. 2024. Apple Intelligence Comes to iPhone iPad and Mac starting next month. https:\/\/www.apple.com\/de\/newsroom\/2024\/09\/apple-intelligence-comes-to-iphone-ipad-and-mac-starting-next-month\/. [Accessed 20-09-2024]."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-008-0001-3"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3654777.3676372"},{"key":"e_1_2_1_8_1","volume-title":"Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432","author":"Bengio Yoshua","year":"2013","unstructured":"Yoshua Bengio, Nicholas L\u00e9onard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432 (2013)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3569466"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544549.3585672"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589132.3625625"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/0-387-33006-2_4"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Geoff Boeing. 2024. Modeling and Analyzing Urban Networks and Amenities with OSMnx. (2024).","DOI":"10.2139\/ssrn.5236246"},{"key":"e_1_2_1_14_1","volume-title":"Using thematic analysis in psychology. Qualitative research in psychology 3, 2","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101."},{"key":"e_1_2_1_15_1","volume-title":"Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving. 2024 IEEE International Conference on Robotics and Automation (ICRA)","author":"Chen Long","year":"2024","unstructured":"Long Chen, Oleg Sinavski, Jan H\u00fcnermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. 2024. Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving. 2024 IEEE International Conference on Robotics and Automation (ICRA) (2024), 14093--14100."},{"key":"e_1_2_1_16_1","unstructured":"Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168 (2021)."},{"key":"e_1_2_1_17_1","volume-title":"International Conference on Learning Representations (ICLR).","author":"Dao Tri","year":"2024","unstructured":"Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_1_18_1","first-page":"18090","article-title":"Pengi: An Audio Language Model for Audio Tasks","volume":"36","author":"Deshmukh Soham","year":"2023","unstructured":"Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An Audio Language Model for Audio Tasks. Advances in Neural Information Processing Systems 36 (2023), 18090--18108.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_19_1","volume-title":"QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems 36","author":"Dettmers Tim","year":"2024","unstructured":"Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 24th international conference on intelligent user interfaces. 528--537","author":"Frison Anna-Katharina","year":"2019","unstructured":"Anna-Katharina Frison, Philipp Wintersberger, Tianjia Liu, and Andreas Riener. 2019. Why do you like to drive automated? a context-dependent analysis of highly automated driving to elaborate requirements for intelligent user interfaces. In Proceedings of the 24th international conference on intelligent user interfaces. 528--537."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00509"},{"key":"e_1_2_1_22_1","volume-title":"Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.","author":"Hendrycks Dan","year":"2021","unstructured":"Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations."},{"key":"e_1_2_1_23_1","unstructured":"Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Dawn Song and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving with the MATH Dataset. In Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https:\/\/openreview.net\/forum?id=7Bywt2mQsCe"},{"key":"e_1_2_1_24_1","unstructured":"John Hewitt. 2021. Initializing New Word Embeddings for Pretrained Language Models. https:\/nlp.stanford.edu\/~johnhew\/\/vocab-expansion.html."},{"key":"e_1_2_1_25_1","volume-title":"A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics","author":"Holm Sture","year":"1979","unstructured":"Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65--70."},{"key":"e_1_2_1_26_1","volume-title":"LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations.","author":"Hu Edward J","year":"2022","unstructured":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations."},{"key":"e_1_2_1_27_1","volume-title":"Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.","author":"Jiang Albert Q","year":"2023","unstructured":"Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)."},{"key":"e_1_2_1_28_1","volume-title":"MotionGPT: Human Motion as a Foreign Language. Advances in Neural Information Processing Systems 36","author":"Jiang Biao","year":"2024","unstructured":"Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2024. MotionGPT: Human Motion as a Foreign Language. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_2_1_29_1","volume-title":"Soundsride: Affordance-Synchronized Music Mixing for In-Car Audio Augmented Reality. In The 34th Annual ACM Symposium on User Interface Software and Technology. 118--133","author":"Kari Mohamed","year":"2021","unstructured":"Mohamed Kari, Tobias Grosse-Puppendahl, Alexander Jagaciak, David Bethge, Reinhard Sch\u00fctte, and Christian Holz. 2021. Soundsride: Affordance-Synchronized Music Mixing for In-Car Audio Augmented Reality. In The 34th Annual ACM Symposium on User Interface Software and Technology. 118--133."},{"key":"e_1_2_1_30_1","unstructured":"Eryk Lewinson. 2022. Three Approaches to Encoding Time Information as Features for ML Models. NVIDIA Developer Blog. Accessed 2024-09-02."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11771"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.317"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.02484"},{"key":"e_1_2_1_34_1","volume-title":"Visual Instruction Tuning. Advances in Neural Information Processing Systems 36","author":"Liu Haotian","year":"2024","unstructured":"Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual Instruction Tuning. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_2_1_35_1","volume-title":"The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=TqL2xBwXP3","author":"Manvi Rohin","year":"2024","unstructured":"Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David B. Lobell, and Stefano Ermon. 2024. GeoLLM: Extracting Geospatial Knowledge from Large Language Models. In The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=TqL2xBwXP3"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 89th Annual Meeting of the Transportation Research Board on Traffic and Transport Planning. 1--15","author":"Mehler Bruce","year":"2010","unstructured":"Bruce Mehler, Bryan Reimer, Lisa A D'Ambrosio, Alexander Pi\u00f1a, and Joseph F Coughlin. 2010. An Evaluation of Time of Day Influences on Simulated Driving Performance and Physiological Arousal. In Proceedings of the 89th Annual Meeting of the Transportation Research Board on Traffic and Transport Planning. 1--15."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3615886.3627745"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0028"},{"key":"e_1_2_1_39_1","volume-title":"International Conference on Machine Learning, 8748--8763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning Transferable Visual Models from Natural Language Supervision. International Conference on Machine Learning, 8748--8763."},{"key":"e_1_2_1_40_1","volume-title":"Aaron Van den Oord, and Oriol Vinyals","author":"Razavi Ali","year":"2019","unstructured":"Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems 32 (2019)."},{"key":"e_1_2_1_41_1","volume-title":"GPT4GEO: How a Language Model Sees the World's Geography. arXiv preprint arXiv:2306.00020","author":"Roberts Jonathan","year":"2023","unstructured":"Jonathan Roberts, Timo L\u00fcddecke, Sowmen Das, Kai Han, and Samuel Albanie. 2023. GPT4GEO: How a Language Model Sees the World's Geography. arXiv preprint arXiv:2306.00020 (2023)."},{"key":"e_1_2_1_42_1","volume-title":"Variational analysis","author":"Tyrrell Rockafellar R","unstructured":"R Tyrrell Rockafellar and Roger J-B Wets. 2009. Variational analysis. Vol. 317. Springer Science & Business Media."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474381"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300867"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3631404"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_2_1_47_1","volume-title":"A Multimodal Approach for Monitoring Driving Behavior and Emotions","author":"Tavakoli Arash","unstructured":"Arash Tavakoli, Vahid Balali, and Arsalan Heydarian. 2020. A Multimodal Approach for Monitoring Driving Behavior and Emotions. Mineta Transportation Institute."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-50943-9_5"},{"key":"e_1_2_1_49_1","unstructured":"Aaron Van Den Oord Oriol Vinyals et al. 2017. Neural Discrete Representation Learning. Advances in Neural Information Processing Systems 30 (2017)."},{"key":"e_1_2_1_50_1","article-title":"Visualizing Data Using t-SNE","volume":"9","author":"der Maaten Laurens Van","year":"2008","unstructured":"Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing Data Using t-SNE. Journal of Machine Learning Research 9, 11 (2008).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_51_1","first-page":"24824","article-title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (2022), 24824--24837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38--45","author":"Wolf Thomas","year":"2020","unstructured":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R\u00e9mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, Online, 38--45. https:\/\/www.aclweb.org\/anthology\/2020.emnlp-demos.6"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3478125"},{"key":"e_1_2_1_54_1","volume-title":"A Survey on Multimodal Large Language Models. arXiv preprint arXiv:2306.13549","author":"Yin Shukang","year":"2023","unstructured":"Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A Survey on Multimodal Large Language Models. arXiv preprint arXiv:2306.13549 (2023)."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1472"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-demo.49"},{"key":"e_1_2_1_57_1","unstructured":"Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric Xing et al. 2024. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (2024)."}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3749552","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,25]],"date-time":"2025-09-25T16:27:04Z","timestamp":1758817624000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3749552"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,3]]},"references-count":57,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,3]]}},"alternative-id":["10.1145\/3749552"],"URL":"https:\/\/doi.org\/10.1145\/3749552","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,3]]},"assertion":[{"value":"2025-09-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}