{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T14:45:40Z","timestamp":1778251540036,"version":"3.51.4"},"reference-count":47,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,12,17]],"date-time":"2025-12-17T00:00:00Z","timestamp":1765929600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Science and Technology Projects of State Grid Jiangsu Electric Power Company Ltd.","award":["J2024168"],"award-info":[{"award-number":["J2024168"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Building high-quality multimodal instruction datasets is often time-consuming and costly. Recent studies have shown that a small amount of carefully selected high-quality data can be more effective for improving LVLM performance than large volumes of low-quality data. Based on these observations, we propose an error-guided multimodal sample selection framework with hallucination suppression for LVLM fine-tuning. First, semantic embeddings of queries are clustered to form balanced subsets that preserve task diversity. A visual contrastive decoding module is then used to reduce hallucinations and expose genuinely difficult examples. For closed-ended tasks, such as object detection, we estimate sample value using prediction accuracy; for open-ended question answering, we use the perplexity of generated responses as a difficulty signal. Within each cluster, high-error or high-perplexity samples are preferentially selected to construct a compact yet informative training set. Experiments on the InsPLAD detection benchmark and the PowerQA visual question answering dataset show that our method consistently outperforms random sampling under the same data budget, achieving higher F1, cosine similarity, BLEU (Bilingual Evaluation Understudy), and GPT-4o-based evaluation scores. This demonstrates that hallucination-aware, uncertainty-driven data selection can improve LVLM robustness and data efficiency.<\/jats:p>","DOI":"10.3390\/computers14120564","type":"journal-article","created":{"date-parts":[[2025,12,17]],"date-time":"2025-12-17T13:42:52Z","timestamp":1765978972000},"page":"564","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Error-Guided Multimodal Sample Selection with Hallucination Suppression for LVLMs"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3287-7109","authenticated-orcid":false,"given":"Huanyu","family":"Cheng","sequence":"first","affiliation":[{"name":"Information and Communication Branch, State Grid Jiangsu Electric Power Company Ltd., Nanjing 210024, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Linjiang","family":"Shang","sequence":"additional","affiliation":[{"name":"Information and Communication Branch, State Grid Jiangsu Electric Power Company Ltd., Nanjing 210024, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xikang","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tao","family":"Feng","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yin","family":"Zhang","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,12,17]]},"reference":[{"key":"ref_1","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv."},{"key":"ref_2","unstructured":"Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., and Rosen, E. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv."},{"key":"ref_3","unstructured":"Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., and Tang, J. (2025). Qwen2.5-vl technical report. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Liu, H., Li, C., Li, Y., and Lee, Y.J. (2024, January 16\u201322). Improved baselines with visual instruction tuning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02484"},{"key":"ref_5","unstructured":"Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023). Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv."},{"key":"ref_6","unstructured":"Xu, H., Ye, Q., Yan, M., Shi, Y., Ye, J., Xu, Y., Li, C., Bi, B., Qian, Q., and Wang, W. (2023, January 23\u201329). mplug-2: A modularized multi-modal foundation model across text, image and video. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA."},{"key":"ref_7","first-page":"49250","article-title":"Instructblip: Towards general-purpose vision-language models with instruction tuning","volume":"36","author":"Dai","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_8","unstructured":"Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., and Shao, J. (2025). Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv."},{"key":"ref_9","unstructured":"Zhang, Y.F., Yu, T., Tian, H., Fu, C., Li, P., Zeng, J., Xie, W., Shi, Y., Zhang, H., and Wu, J. (2025). Mm-rlhf: The next step forward in multimodal llm alignment. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yu, T., Zhang, H., Li, Q., Xu, Q., Yao, Y., Chen, D., Lu, X., Cui, G., Dang, Y., and He, T. (2025, January 11\u201315). Rlaif-v: Open-source AI feedback leads to super gpt-4v trustworthiness. Proceedings of the Computer Vision and Pattern Recognition Conference, Nashville, TN, USA.","DOI":"10.1109\/CVPR52734.2025.01861"},{"key":"ref_11","unstructured":"Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. (2023). Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv."},{"key":"ref_12","unstructured":"Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. (2024). mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv."},{"key":"ref_13","unstructured":"Zhao, B., Wu, B., He, M., and Huang, T. (2023). Svit: Scaling up visual instruction tuning. arXiv."},{"key":"ref_14","unstructured":"Chen, J., Zhang, T., Liu, C., Ding, H., Shi, Y., Cheng, F., Xiao, H., Wen, B., Yang, F., and Gao, T. (2025). Taskgalaxy: Scaling multi-modal instruction fine-tuning with tens of thousands vision task types. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Liu, J., Huang, X., Zheng, J., Liu, B., Wang, J., Yoshie, O., Liu, Y., and Li, H. (2024). Mm-instruct: Generated visual instructions for large multimodal model alignment. arXiv.","DOI":"10.1016\/j.neucom.2025.131164"},{"key":"ref_16","unstructured":"Wang, J., Meng, L., Weng, Z., He, B., Wu, Z., and Jiang, Y.G. (2023). To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv."},{"key":"ref_17","first-page":"55006","article-title":"Lima: Less is more for alignment","volume":"36","author":"Zhou","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Liu, Z., Zhou, K., Zhao, W.X., Gao, D., Li, Y., and Wen, J.R. (2024). Less is more: Data value estimation for visual instruction tuning. arXiv.","DOI":"10.1145\/3746027.3755160"},{"key":"ref_19","unstructured":"Wu, S., Lu, K., Xu, B., Lin, J., Su, Q., and Zhou, C. (2023). Self-evolved diverse data sampling for efficient instruction tuning. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, Y., Hui, B., Xia, X., Yang, J., Yang, M., Zhang, L., Si, S., Chen, L.H., Liu, J., and Liu, T. (2023). One-shot learning as instruction data prospector for large language models. arXiv.","DOI":"10.18653\/v1\/2024.acl-long.252"},{"key":"ref_21","unstructured":"Wei, L., Jiang, Z., Huang, W., and Sun, L. (2023). Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv."},{"key":"ref_22","unstructured":"Cao, Y., Kang, Y., and Sun, L. (2023). Instruction mining: High-quality instruction data selection for large language models. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. (2023). From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv.","DOI":"10.18653\/v1\/2024.naacl-long.421"},{"key":"ref_24","unstructured":"Du, Q., Zong, C., and Zhang, J. (2023). MoDS: Model-oriented Data Selection for Instruction Tuning. arXiv."},{"key":"ref_25","unstructured":"Bi, J., Wang, Y., Yan, D., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. (2025). Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. arXiv."},{"key":"ref_26","first-page":"83465","article-title":"Smalltolarge (s2l): Scalable data selection for fine-tuning large language models by summarizing training trajectories of small models","volume":"37","author":"Yang","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","unstructured":"Yu, Q., Shen, Z., Yue, Z., Wu, Y., Qin, B., Zhang, W., Li, Y., Li, J., Tang, S., and Zhuang, Y. (2024). Mastering collaborative multi-modal data selection: A focus on informativeness, uniqueness, and representativeness. arXiv."},{"key":"ref_28","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv."},{"key":"ref_29","unstructured":"Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., and Bi, X. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv."},{"key":"ref_30","unstructured":"Hong, W., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H., Cheng, J., Qi, J., Ji, J., and Pan, L. (2025). Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv."},{"key":"ref_31","unstructured":"Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., and Ge, W. (2024). Qwen2-vl: Enhancing vision-language model\u2019s perception of the world at any resolution. arXiv."},{"key":"ref_32","unstructured":"Team, K., Du, A., Yin, B., Xing, B., Qu, B., Wang, B., Chen, C., Zhang, C., Du, C., and Wei, C. (2025). Kimi-vl technical report. arXiv."},{"key":"ref_33","unstructured":"Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., and He, Z. (2024). Minicpm-v: A gpt-4v level mllm on your phone. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Hong, W., Wang, W., Ding, M., Yu, W., Lv, Q., Wang, Y., Cheng, Y., Huang, S., Ji, J., and Xue, Z. (2024). Cogvlm2: Visual language models for image and video understanding. arXiv.","DOI":"10.1109\/CVPR52733.2024.01354"},{"key":"ref_35","unstructured":"Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.S. (2024, January 21\u201327). Next-gpt: Any-to-any multimodal llm. Proceedings of the Forty-first International Conference on Machine Learning, Vienna, Austria."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"nwae403","DOI":"10.1093\/nsr\/nwae403","article-title":"A survey on multimodal large language models","volume":"11","author":"Yin","year":"2024","journal-title":"Natl. Sci. Rev."},{"key":"ref_37","unstructured":"Nguyen, T.T., Wilson, C., and Dalins, J. (2025). Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., and Hajishirzi, H. (2022). Self-instruct: Aligning language models with self-generated instructions. arXiv.","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"ref_39","unstructured":"Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. (2024, January 7\u201311). WizardLM: Empowering large pre-trained language models to follow complex instructions. Proceedings of the The Twelfth International Conference on Learning Representations, Vienna, Austria."},{"key":"ref_40","unstructured":"Zhao, Y., Yu, B., Hui, B., Yu, H., Huang, F., Li, Y., and Zhang, N.L. (2024). A preliminary study of the intrinsic relationship between complexity and alignment. arXiv."},{"key":"ref_41","unstructured":"Shengyu, Z., Linfeng, D., Xiaoya, L., Sen, Z., Xiaofei, S., Shuhe, W., Jiwei, L., Hu, R., Tianwei, Z., and Wu, F. (2023). Instruction tuning for large language models: A survey. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Li, M., Zhang, Y., He, S., Li, Z., Zhao, H., Wang, J., Cheng, N., and Zhou, T. (2024). Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. arXiv.","DOI":"10.18653\/v1\/2024.acl-long.769"},{"key":"ref_43","unstructured":"Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., and Huang, H. (2023). Alpagasus: Training a better alpaca with fewer data. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., and Bing, L. (2024, January 16\u201322). Mitigating object hallucinations in large vision-language models through visual contrastive decoding. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.01316"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"7294","DOI":"10.1080\/01431161.2023.2283900","article-title":"Insplad: A dataset and benchmark for power line asset inspection in uav images","volume":"44","author":"Teichrieb","year":"2023","journal-title":"Int. J. Remote Sens."},{"key":"ref_46","first-page":"3","article-title":"LoRa: Low-rank adaptation of large language models","volume":"1","author":"Hu","year":"2022","journal-title":"ICLR"},{"key":"ref_47","unstructured":"Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. (2023). Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/12\/564\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,17]],"date-time":"2025-12-17T13:48:49Z","timestamp":1765979329000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/12\/564"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,17]]},"references-count":47,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["computers14120564"],"URL":"https:\/\/doi.org\/10.3390\/computers14120564","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,17]]}}}