{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T19:53:12Z","timestamp":1783021992495,"version":"3.54.6"},"reference-count":54,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T00:00:00Z","timestamp":1719273600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202495"],"award-info":[{"award-number":["52202495"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202494"],"award-info":[{"award-number":["52202494"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human-level driving is the ultimate goal of autonomous driving. As the top-level decision-making aspect of autonomous driving, behavior decision establishes short-term driving behavior strategies by evaluating road structures, adhering to traffic rules, and analyzing the intentions of other traffic participants. Existing behavior decisions are primarily implemented based on rule-based methods, exhibiting insufficient generalization capabilities when faced with new and unseen driving scenarios. In this paper, we propose a novel behavior decision method that leverages the inherent generalization and commonsense reasoning abilities of visual language models (VLMs) to learn and simulate the behavior decision process in human driving. We constructed a novel instruction-following dataset containing a large number of image\u2013text instructions paired with corresponding driving behavior labels, to support the learning of the Drive Large Language and Vision Assistant (DriveLLaVA) and enhance the transparency and interpretability of the entire decision process. DriveLLaVA is fine-tuned on this dataset using the Low-Rank Adaptation (LoRA) approach, which efficiently optimizes the model parameter count and significantly reduces training costs. We conducted extensive experiments on a large-scale instruction-following dataset, and compared with state-of-the-art methods, DriveLLaVA demonstrated excellent behavior decision performance. DriveLLaVA is capable of handling various complex driving scenarios, showing strong robustness and generalization abilities.<\/jats:p>","DOI":"10.3390\/s24134113","type":"journal-article","created":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T09:29:33Z","timestamp":1719394173000},"page":"4113","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["DriveLLaVA: Human-Level Behavior Decisions via Vision Language Model"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1597-1961","authenticated-orcid":false,"given":"Rui","family":"Zhao","sequence":"first","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qirui","family":"Yuan","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8463-7732","authenticated-orcid":false,"given":"Jinyu","family":"Li","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8309-7396","authenticated-orcid":false,"given":"Yuze","family":"Fan","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7824-7751","authenticated-orcid":false,"given":"Yun","family":"Li","sequence":"additional","affiliation":[{"name":"Graduate School of Information and Science Technology, The University of Tokyo, Tokyo 113-8654, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4195-5033","authenticated-orcid":false,"given":"Fei","family":"Gao","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"},{"name":"National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/MRA.2020.3045040","article-title":"The role of the hercules autonomous vehicle during the covid-19 pandemic: An autonomous logistic vehicle for contactless goods transportation","volume":"28","author":"Liu","year":"2021","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Parekh, D., Poddar, N., and Rajpurkar, A. (2022). A review on autonomous vehicles: Progress, methods and challenges. Electronics, 11.","DOI":"10.3390\/electronics11142162"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"26543","DOI":"10.1109\/ACCESS.2019.2900416","article-title":"A novel lane change decision-making model of autonomous vehicle based on support vector machine","volume":"7","author":"Liu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_4","first-page":"20","article-title":"A Rule-Based Expert System for Automobile Fault Diagnosis","volume":"7","author":"Ahmad","year":"2021","journal-title":"Int. J. Perceptive Cogn. Comput."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Claussmann, L., O\u2019Brien, M., and Glaser, S. (2018, January 26\u201330). Multi-criteria decision making for autonomous vehicles using fuzzy dempster-shafer reasoning. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500451"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/0001-4575(89)90025-0","article-title":"Explanatory pitfalls and rule-based driver models","volume":"21","author":"Michon","year":"1989","journal-title":"Accid. Anal. Prev."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1007\/s10462-018-9631-5","article-title":"Artificial intelligence test: A case study of intelligent vehicles","volume":"50","author":"Li","year":"2018","journal-title":"Artif. Intell. Rev."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/MITS.2021.3070651","article-title":"Acclimatizing the operational design domain for autonomous driving systems","volume":"14","author":"Sun","year":"2021","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"3995","DOI":"10.1109\/TIE.2022.3177788","article-title":"Personalized lane change planning and control by imitation learning from drivers","volume":"70","author":"Tian","year":"2022","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ozcelik, M.B., Agin, B., and Caldiran, O. (2023, January 11\u201313). Decision Making for Autonomous Driving in a Virtual Highway Environment based on Generative Adversarial Imitation Learning. Proceedings of the 2023 Innovations in Intelligent Systems and Applications Conference (ASYU), Sivas, T\u00fcrkiye.","DOI":"10.1109\/ASYU58738.2023.10296611"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2874","DOI":"10.1109\/TITS.2022.3227738","article-title":"Modeling human driving behavior through generative adversarial imitation learning","volume":"24","author":"Bhattacharyya","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kamran, D., Ren, Y., and Lauer, M. (2021, January 19\u201322). High-level decisions from a safe maneuver catalog with reinforcement learning for safe and cooperative automated merging. Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA.","DOI":"10.1109\/ITSC48978.2021.9564912"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2450","DOI":"10.1109\/TITS.2023.3323440","article-title":"Prediction-Aware and Reinforcement Learning-Based Altruistic Cooperative Driving","volume":"25","author":"Valiente","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"854","DOI":"10.1109\/TITS.2022.3216288","article-title":"Multi-agent DRL-based lane change with right-of-way collaboration awareness","volume":"24","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"24791","DOI":"10.1109\/TITS.2022.3207872","article-title":"Social coordination and altruism in autonomous driving","volume":"23","author":"Toghi","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, P., Liu, D., and Chen, J. (June, January 30). Decision making for autonomous driving via augmented adversarial inverse reinforcement learning. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9560907"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1109\/MIS.2007.41","article-title":"Social computing: From social informatics to social intelligence","volume":"22","author":"Wang","year":"2007","journal-title":"IEEE Intell. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1109\/MITS.2023.3278158","article-title":"Forward to the Past: CASTLab\u2019s Cyber-Social-Physical Approach for ITS in 1999 [History and Perspectives]","volume":"15","author":"Wang","year":"2023","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2025","DOI":"10.1109\/TSMC.2022.3225250","article-title":"Sharing traffic priorities via cyber\u2013physical\u2013social intelligence: A lane-free autonomous intersection management method in metaverse","volume":"53","author":"Li","year":"2022","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_20","first-page":"16","article-title":"Parallel intelligence in metaverses: Welcome to Hanoi!","volume":"37","author":"Wang","year":"2022","journal-title":"IEEE Intell. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"16962","DOI":"10.1109\/TITS.2022.3156011","article-title":"Scenario understanding and motion prediction for autonomous vehicles\u2014Review and comparison","volume":"23","author":"Karle","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1046","DOI":"10.1109\/TIV.2022.3223131","article-title":"Milestones in autonomous driving and intelligent vehicles: Survey of surveys","volume":"8","author":"Chen","year":"2022","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2041","DOI":"10.1109\/JAS.2023.123966","article-title":"The ChatGPT after: Building knowledge factories for knowledge workers with knowledge automation","volume":"10","author":"Wang","year":"2023","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13\u201319). nuScenes: A Multimodal Dataset for Autonomous Driving. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/TITS.2020.3012034","article-title":"Deep learning-based vehicle behavior prediction for autonomous driving applications: A review","volume":"23","author":"Mozaffari","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_26","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_28","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_29","unstructured":"OpenAI (2023). GPT-4 technical report. arXiv."},{"key":"ref_30","first-page":"1","article-title":"Palm: Scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2023","journal-title":"J. Mach. Learn. Res."},{"key":"ref_31","unstructured":"Zheng, L., Chiang, W.L., and Sheng, Y. (2024). Judging llm-as-a-judge with mt-bench and chatbot arena. Adv. Neural Inf. Process. Syst., 36."},{"key":"ref_32","unstructured":"Touvron, H., Lavril, T., and Izacard, G. (2023). Llama: Open and efficient foundation language models. arXiv."},{"key":"ref_33","unstructured":"Touvron, H., Martin, L., and Stone, K. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_34","unstructured":"Radford, A., Kim, J.W., and Hallacy, C. (2021, January 18\u201324). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_35","unstructured":"Driess, D., Xia, F., and Sajjadi, M.S.M. (2023). Palm-e: An embodied multimodal language model. arXiv."},{"key":"ref_36","unstructured":"Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., and Chang, K.W. (2019). VisualBERT: A Simple and Performant Baseline for Vision and Language. arXiv."},{"key":"ref_37","unstructured":"Wang, Z., Yu, J., and Yu, A.W. (2021). Simvlm: Simple visual language model pretraining with weak supervision. arXiv."},{"key":"ref_38","first-page":"23716","article-title":"Flamingo: A visual language model for few-shot learning","volume":"35","author":"Alayrac","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_39","unstructured":"Li, J., Li, D., and Savarese, S. (2023, January 23\u201329). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA."},{"key":"ref_40","unstructured":"Liu, H., Li, C., and Li, Y. (2023). Improved baselines with visual instruction tuning. arXiv."},{"key":"ref_41","unstructured":"Bai, J., Bai, S., and Yang, S. (2023). Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Fu, D., Li, X., and Wen, L. (2024, January 1\u20136). Drive like a human: Rethinking autonomous driving with large language models. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACVW60836.2024.00102"},{"key":"ref_43","unstructured":"Mao, J., Qian, Y., and Zhao, H. (2023). Gpt-driver: Learning to drive with gpt. arXiv."},{"key":"ref_44","unstructured":"Tian, X., Gu, J., and Li, B. (2024). DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models. arXiv."},{"key":"ref_45","unstructured":"Sima, C., Renz, K., and Chitta, K. (2023). Drivelm: Driving with graph visual question answering. arXiv."},{"key":"ref_46","unstructured":"Wu, D., Han, W., and Wang, T. (2023). Language Prompt for Autonomous Driving. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Qian, T., Chen, J., and Zhuo, L. (2024, January 20\u201327). NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i5.28253"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Sachdeva, E., Agarwal, N., and Chundi, S. (2024, January 1\u20136). Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV57701.2024.00734"},{"key":"ref_49","unstructured":"Xu, Z., Zhang, Y., and Xie, E. (2023). DriveGPT4: Interpretable End-to-End Autonomous Driving via Large Language Model. arXiv."},{"key":"ref_50","unstructured":"Movva, R., Balachandar, S., and Peng, K. (2023). Large Language Models Shape and Are Shaped by Society: A Survey of arXiv Publication Patterns. arXiv."},{"key":"ref_51","unstructured":"Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P.N., and Hoi, S. (2024). InstructBLIP: Towards General-Purpose Vision-Language Models with Instruction Tuning. Adv. Neural Inf. Process. Syst., 36."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Xu, K., Xiao, X., Miao, J., and Luo, Q. (November, January 19). Data Driven Prediction Architecture for Autonomous Driving and Its Application on Apollo Platform. Proceedings of the 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA.","DOI":"10.1109\/IV47402.2020.9304810"},{"key":"ref_53","unstructured":"Wang, W., Xie, J., Hu, C.Y., Zou, H., Fan, J., Tong, W., Wen, Y., Wu, S., Deng, H., and Li, Z. (2023). DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Jaeger, B., Chitta, K., and Geiger, A. (2023, January 2\u20136). Hidden Biases of End-to-End Driving Models. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.00757"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/13\/4113\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:03:52Z","timestamp":1760108632000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/13\/4113"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,25]]},"references-count":54,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["s24134113"],"URL":"https:\/\/doi.org\/10.3390\/s24134113","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,25]]}}}