{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T10:48:59Z","timestamp":1767869339806,"version":"3.49.0"},"reference-count":47,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T00:00:00Z","timestamp":1767657600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>Recent vision-language pre-training models, like CLIP, have been shown to generalize well across a variety of multitask modalities. Nonetheless, their generalization for downstream tasks is limited. As a lightweight adaptation approach, prompt learning could allow task transfer by optimizing only several learnable vectors and thus is more flexible for pre-trained models. However, current methods mainly concentrate on the design of unimodal prompts and ignore effective means for multimodal semantic fusion and label alignment, which limits their representation power. To tackle these problems, this paper designs a Hierarchical Interactive Semantic Fusion (HISF) framework for multimodal prompt learning. On top of frozen CLIP backbones, HISF injects visual and textual signals simultaneously in intermediate layers of a Transformer through a cross-attention mechanism as well as fitting category embeddings. This architecture realizes the hierarchical semantic fusion at the modality level with structural consistency kept at each layer. In addition, a Label Embedding Constraint and a Semantic Alignment Loss are proposed to promote category consistency while alleviating semantic drift in training. Extensive experiments across 11 few-shot image classification benchmarks show that HISF improves the average accuracy by around 0.7% compared to state-of-the-art methods and has remarkable robustness in cross-domain transfer tasks. Ablation studies also verify the effectiveness of each proposed part and their combination: hierarchical structure, cross-modal attention, and semantic alignment collaborate to enrich representational capacity. In conclusion, the proposed HISF is a new hierarchical view for multimodal prompt learning and provides a more lightweight and generalizable paradigm for adapting vision-language pre-trained models.<\/jats:p>","DOI":"10.3390\/mti10010006","type":"journal-article","created":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T13:49:46Z","timestamp":1767707386000},"page":"6","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["HISF: Hierarchical Interactive Semantic Fusion for Multimodal Prompt Learning"],"prefix":"10.3390","volume":"10","author":[{"given":"Haohan","family":"Feng","sequence":"first","affiliation":[{"name":"State Key Laboratory of Media Convergence and Communication, School of Information and Communication Engineering, Communication University of China, Beijing 100024, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chen","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Media Convergence and Communication, School of Information and Communication Engineering, Communication University of China, Beijing 100024, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,1,6]]},"reference":[{"key":"ref_1","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18\u201324). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual Event."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1007\/s11633-022-1369-5","article-title":"VLP: A survey on vision-language pre-training","volume":"20","author":"Chen","year":"2023","journal-title":"Mach. Intell. Res."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Gu, Y., Han, X., Liu, Z., and Huang, M. (2022, January 22\u201327). PPT: Pre-trained prompt tuning for few-shot learning. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland.","DOI":"10.18653\/v1\/2022.acl-long.576"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2337","DOI":"10.1007\/s11263-022-01653-1","article-title":"Learning to prompt for vision-language models","volume":"130","author":"Zhou","year":"2022","journal-title":"Int. J. Comput. Vis."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhou, K., Yang, J., Loy, C.C., and Liu, Z. (2022, January 18\u201324). Conditional prompt learning for vision-language models. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01631"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., and Khan, F.S. (2023, January 17\u201324). MaPLe: Multi-modal prompt learning. Proceedings of the 2023 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01832"},{"key":"ref_7","first-page":"1815","article-title":"LAMM: Label alignment for multi-modal prompt learning","volume":"38","author":"Gao","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"130028","DOI":"10.1016\/j.neucom.2025.130028","article-title":"Fine-grained multi-modal prompt learning for vision\u2013language models","volume":"636","author":"Liu","year":"2025","journal-title":"Neurocomputing"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Guo, Y., and Gu, X. (2025, January 11\u201315). MMRL: Multi-modal representation learning for vision-language models. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR52734.2025.02329"},{"key":"ref_10","unstructured":"Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., and Luo, P. (December, January 28). AdaptFormer: Adapting vision transformers for scalable visual recognition. Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Du, Y., Liu, Z., Li, J., and Zhao, W.X. (2022, January 23\u201329). A survey of vision-language pre-trained models. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI 2022), Vienna, Austria.","DOI":"10.24963\/ijcai.2022\/762"},{"key":"ref_12","unstructured":"Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Jitsev, J., and Komatsuzaki, A. (2021, January 6\u201314). LAION-400M: Open dataset of clip-filtered 400 million image-text pairs. Proceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS 2021), Virtual Event."},{"key":"ref_13","unstructured":"Yang, A., Pan, J., Lin, J., Men, R., Zhang, J., Zhou, J., and Zhou, C. (2022). Chinese CLIP: Contrastive vision-language pretraining in Chinese. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020, January 23\u201328). OSCAR: Object-semantics aligned pre-training for vision-language tasks. Proceedings of the 16th European Conference on Computer Vision\u2014ECCV 2020, Glasgow, UK.","DOI":"10.1007\/978-3-030-58577-8_8"},{"key":"ref_15","unstructured":"Li, J., Li, D., Savarese, S., and Hoi, S. (2023, January 23\u201329). BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA."},{"key":"ref_16","unstructured":"Liu, H., Li, C., Wu, Q., and Lee, Y.J. (2023, January 10\u201316). Visual instruction tuning. Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18\u201323). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00636"},{"key":"ref_18","unstructured":"Jia, C., Yang, Y., Xia, Y., Chen, Y., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., and Duerig, T. (2021, January 18\u201324). Scaling up visual and vision-language representation learning with noisy text supervision. Proceedings of the 38th International Conference on Machine Learning, Virtual Event."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. (2022, January 18\u201324). LIT: Zero-shot transfer with locked-image text tuning. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVP), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01759"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1007\/s11263-023-01891-x","article-title":"CLIP-Adapter: Better vision-language models with feature adapters","volume":"132","author":"Gao","year":"2024","journal-title":"Int. J. Comput. Vis."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Cho, J., Nam, G., Kim, S., Yang, H., and Kwak, S. (2023, January 1\u20136). PromptStyler: Prompt-driven style generation for source-free domain generalization. Proceedings of the 2023 IEEE\/CVF International Conference on Computer Vision (ICCV), Paris, France.","DOI":"10.1109\/ICCV51070.2023.01439"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. (2022, January 18\u201324). Prompt distribution learning. Proceedings of the 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00514"},{"key":"ref_23","unstructured":"Sun, X., Hu, P., and Saenko, K. (December, January 28). DualCoOp: Fast adaptation to multi-label recognition with limited annotations. Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_24","first-page":"8718","article-title":"TextRefiner: Internal visual feature as efficient refiner for vision-language models prompt tuning","volume":"39","author":"Xie","year":"2025","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.aiopen.2024.01.004","article-title":"CPT: Colorful prompt tuning for pre-trained vision-language models","volume":"5","author":"Yao","year":"2024","journal-title":"AI Open"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"131253","DOI":"10.1016\/j.neucom.2025.131253","article-title":"Text-guided visual prompt learning with semantic prompt generation and feature fusion","volume":"654","author":"Zhang","year":"2025","journal-title":"Neurocomputing"},{"key":"ref_27","unstructured":"Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. (2022). Exploring visual prompts for adapting large-scale models. arXiv."},{"key":"ref_28","unstructured":"Roy, S., and Etemad, A. (2023). Consistency-guided prompt learning for vision-language models. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhu, J., Ruan, Y., Chang, J., Sun, W., Wan, H., Long, J., and Luo, C. (2024, January 1\u20135). Deep prompt multi-task network for abuse language detection. Proceedings of the 27th International Conference on Pattern Recognition, Kolkata, India.","DOI":"10.1007\/978-3-031-78107-0_16"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.Y., Ren, X., Su, G., Perot, V., and Dy, J. (2022, January 23\u201327). DualPrompt: Complementary prompting for rehearsal-free continual learning. Proceedings of the 17th European Conference on Computer Vision\u2014ECCV 2022, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-19809-0_36"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"102919","DOI":"10.1016\/j.inffus.2024.102919","article-title":"Multi-modal prompt learning with bidirectional layer-wise prompt fusion","volume":"117","author":"Yin","year":"2025","journal-title":"Inf. Fusion"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1145\/3649447","article-title":"Deep Multimodal Data Fusion","volume":"56","author":"Zhao","year":"2024","journal-title":"ACM Comput. Surv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, Y., Quan, R., Zhu, L., and Yang, Y. (2023, January 17\u201324). Efficient multimodal fusion via interactive prompting. Proceedings of the 2023 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00256"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yang, L., Zhang, R., Wang, Y., and Xie, X. (2024, January 16\u201322). MMA: Multi-modal adapter for vision-language models. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02249"},{"key":"ref_35","unstructured":"Wu, Q., Yu, W., Zhou, Y., Huang, S., Sun, X., and Ji, R. (2023, January 10\u201316). Parameter and computation efficient transfer learning for vision-language pre-trained models. Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA."},{"key":"ref_36","first-page":"16076","article-title":"MmAP: Multi-modal alignment prompt for cross-domain multi-task learning","volume":"38","author":"Xin","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"\u00d6zdemir, \u00d6., and Akag\u00fcnd\u00fcz, E. (2024, January 17\u201318). Enhancing visual question answering through question-driven image captions as prompts. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW63382.2024.00163"},{"key":"ref_38","first-page":"5749","article-title":"Learning hierarchical prompt with structured linguistic knowledge for vision-language models","volume":"38","author":"Wang","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3746","DOI":"10.1007\/s11263-025-02354-1","article-title":"Learning with enriched inductive biases for vision-language models","volume":"133","author":"Yang","year":"2025","journal-title":"Int. J. Comput. Vis."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"113011","DOI":"10.1016\/j.knosys.2025.113011","article-title":"A slim prompt-averaged consistency prompt learning for vision\u2013language model","volume":"310","author":"He","year":"2025","journal-title":"Knowl. Based Syst."},{"key":"ref_41","unstructured":"Wang, G., Ge, Y., Ding, X., Kankanhalli, M., and Shan, Y. (2023). What makes for good visual tokenizers for large language models?. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Li, Z., Li, X., Fu, X., Zhang, X., Wang, W., Chen, S., and Yang, J. (2024, January 16\u201322). PromptKD: Unsupervised prompt distillation for vision-language models. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02513"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1348","DOI":"10.1109\/TIP.2024.3362062","article-title":"Learning domain invariant prompt for vision-language models","volume":"33","author":"Zhao","year":"2024","journal-title":"IEEE Trans. Image Process."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1007\/s11263-024-02172-x","article-title":"Progressive visual prompt learning with contrastive feature re-formation","volume":"133","author":"Xu","year":"2025","journal-title":"Int. J. Comput. Vis."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Yao, H., Zhang, R., and Xu, C. (2023, January 17\u201324). Visual-language prompt tuning with knowledge-guided context optimization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00653"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Khattak, M.U., Wasim, S.T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F.S. (2023, January 2\u20136). Self-regulating prompts: Foundational model adaptation without forgetting. Proceedings of the 2023 IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.01394"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Yao, H., Zhang, R., and Xu, C. (2024, January 17\u201321). TCP: Textual-based class-aware prompt tuning for visual-language model. Proceedings of the 2024 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02212"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/10\/1\/6\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T05:16:13Z","timestamp":1767849373000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/10\/1\/6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,6]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,1]]}},"alternative-id":["mti10010006"],"URL":"https:\/\/doi.org\/10.3390\/mti10010006","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,6]]}}}