{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T10:12:24Z","timestamp":1772446344316,"version":"3.50.1"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"12","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62472092, 62172089, and 62106045"],"award-info":[{"award-number":["62472092, 62172089, and 62106045"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004608","name":"Natural Science Foundation of Jiangsu Province","doi-asserted-by":"crossref","award":["BK20241751"],"award-info":[{"award-number":["BK20241751"]}],"id":[{"id":"10.13039\/501100004608","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Jiangsu Provincial Key Laboratory of Computer Networking Technology, Jiangsu Provincial Key Laboratory of Network and Information Security","award":["BM2003201"],"award-info":[{"award-number":["BM2003201"]}]},{"name":"Key Laboratory of Computer Network and Information Integration of Ministry of Education of China","award":["93K-9"],"award-info":[{"award-number":["93K-9"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>\n                    Identifying labels that are unseen during training, known as multi-label zero-shot learning, is a non-trivial task in computer vision. Recent studies have increasingly focused on utilizing vision-language pre-training (VLP) models to recognize unseen labels in an open-vocabulary manner. However, these approaches like knowledge distillation have offered only modest performance gains. The challenge of fully harnessing the potential of VLP models for effective multi-label zero-shot learning remains open. In this work, an advanced query-based knowledge sharing framework is proposed to explore the multi-modal knowledge from VLP models for open-vocabulary multi-label classification. Specifically, we introduce a set of label-agnostic query tokens that are designed to capture essential and informative visual knowledge from input images. These tokens are subsequently shared across all labels, allowing them to select pertinent one as visual clues for accurate recognition. Then, by integrating the pre-trained knowledge of VLP models, these query tokens, trained on seen labels, can be efficiently generalized to the recognition of unseen labels. Additionally, we reformulate ranking learning into a form of classification to enable the magnitude of feature vectors for prediction, which significantly benefits label recognition. Experiment results show that our framework outperforms state-of-the-art methods in multi-label zero-shot learning task by a significant margin, reaching 4.2% and 2.4% in mAP on the NUS-WIDE and Open Images datasets, respectively. Code and models are available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/jasonseu\/QKS\">https:\/\/github.com\/jasonseu\/QKS<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1145\/3762195","type":"journal-article","created":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T16:02:13Z","timestamp":1755619333000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Query-Based Knowledge Sharing for Open-Vocabulary Multi-Label Classification"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7676-2843","authenticated-orcid":false,"given":"Xuelin","family":"Zhu","sequence":"first","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5323-5343","authenticated-orcid":false,"given":"Jian","family":"Liu","sequence":"additional","affiliation":[{"name":"Ant Group Co., Ltd., Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6311-2550","authenticated-orcid":false,"given":"Dongqi","family":"Tang","sequence":"additional","affiliation":[{"name":"Ant Group Co., Ltd., Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7268-7815","authenticated-orcid":false,"given":"Jiawei","family":"Ge","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing,\u00a0China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2634-7283","authenticated-orcid":false,"given":"Weijia","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing,\u00a0China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5209-9063","authenticated-orcid":false,"given":"Bo","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Southeast University, Nanjing, China and Purple Mountain Laboratories, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2448-6717","authenticated-orcid":false,"given":"Jiuxin","family":"Cao","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Engineering, Southeast University, Nanjing, China and Purple Mountain Laboratories, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,11,21]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"32897","article-title":"VLMo: Unified vision-language pre-training with mixture-of-modality-experts","volume":"35","author":"Bao Hangbo","year":"2022","unstructured":"Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. VLMo: Unified vision-language pre-training with mixture-of-modality-experts. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 32897\u201332912.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00068"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00061"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/1646396.1646452"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICME55011.2023.00365"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01369"},{"key":"e_1_3_2_8_2","unstructured":"Yanwei Fu Yongxin Yang Tim Hospedales Tao Xiang and Shaogang Gong. 2015. Transductive multi-label zero-shot learning. arXiv:1503.07790. Retrieved from https:\/\/arxiv.org\/abs\/1503.07790"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-023-01891-x"},{"key":"e_1_3_2_10_2","unstructured":"Golnaz Ghiasi Xiuye Gu Yin Cui and Tsung-Yi Lin. 2021. Open-vocabulary image segmentation. arXiv:2112.12143. Retrieved from https:\/\/arxiv.org\/abs\/2112.12143"},{"key":"e_1_3_2_11_2","unstructured":"Yunchao Gong Yangqing Jia Thomas Leung Alexander Toshev and Sergey Ioffe. 2013. Deep convolutional ranking for multilabel image annotation. arXiv:1312.4894. Retrieved from https:\/\/arxiv.org\/abs\/1312.4894"},{"key":"e_1_3_2_12_2","unstructured":"Xiuye Gu Tsung-Yi Lin Weicheng Kuo and Yin Cui. 2021. Open-vocabulary object detection via vision and language knowledge distillation. arXiv:2104.13921. Retrieved from https:\/\/arxiv.org\/abs\/2104.13921"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3425547"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2020.2984091"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00275"},{"key":"e_1_3_2_16_2","unstructured":"Akshita Gupta Sanath Narayan Salman Khan Fahad Shahbaz Khan Ling Shao and Joost van de Weijer. 2021. Generative multi-label zero-shot learning. arXiv:2101.11606. Retrieved from https:\/\/arxiv.org\/abs\/2101.11606"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i1.25159"},{"key":"e_1_3_2_18_2","unstructured":"Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv:1503.02531. Retrieved from https:\/\/arxiv.org\/abs\/1503.02531"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00880"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.00689"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.01343"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01394"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.473"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-020-01316-z"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00170"},{"key":"e_1_3_2_26_2","first-page":"19730","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Li Junnan","year":"2023","unstructured":"Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning. PMLR, 19730\u201319742."},{"key":"e_1_3_2_27_2","first-page":"12888","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Li Junnan","year":"2022","unstructured":"Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning. PMLR, 12888\u201312900."},{"key":"e_1_3_2_28_2","first-page":"9694","article-title":"Align before fuse: Vision and language representation learning with momentum distillation","volume":"34","author":"Li Junnan","year":"2021","unstructured":"Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 34, 9694\u20139705.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_29_2","unstructured":"Liunian Harold Li Mark Yatskar Da Yin Cho-Jui Hsieh and Kai-Wei Chang. 2019. VisualBERT: A simple and performant baseline for vision and language. arXiv:1908.03557. Retrieved from https:\/\/arxiv.org\/abs\/1908.03557"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767773"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.553"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i4.28139"},{"key":"e_1_3_2_33_2","unstructured":"Shilong Liu Lei Zhang Xiao Yang Hang Su and Jun Zhu. 2021. 2021. Query2Label: A simple transformer way to multi-label classification. arXiv:2107.10834. Retrieved from https:\/\/arxiv.org\/abs\/2107.10834"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i18.34098"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2022.3222657"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.02285"},{"key":"e_1_3_2_37_2","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101. Retrieved from https:\/\/arxiv.org\/abs\/1711.05101"},{"key":"e_1_3_2_38_2","first-page":"3409","article-title":"A transferable generative framework for multi-label zero-shot learning","author":"Ma Peirong","year":"2023","unstructured":"Peirong Ma, Zhiquan He, Wu Ran, and Hong Lu. 2023. A transferable generative framework for multi-label zero-shot learning. IEEE Transactions on Circuits and Systems for Video Technology 34, 5 (2023), 3409\u20133423.","journal-title":"IEEE Transactions on Circuits and Systems for Video Technology"},{"key":"e_1_3_2_39_2","first-page":"14074","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Ma Zongyang","year":"2022","unstructured":"Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. 2022. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 14074\u201314083."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2018.2848458"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00861"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_44_2","first-page":"8748","volume-title":"Proceedings of the International Conference on Machine Learning. PMLR","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning. PMLR, 8748\u20138763."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2019.2924511"},{"key":"e_1_3_2_46_2","article-title":"Multiple instance visual-semantic embedding","author":"Ren Zhou","year":"2017","unstructured":"Zhou Ren, Hailin Jin, Zhe Lin, Chen Fang, and Alan L. Yuille. 2017. Multiple instance visual-semantic embedding. In Proceedings of the British Machine Vision Conference (BMVC).","journal-title":"Proceedings of the British Machine Vision Conference (BMVC)"},{"key":"e_1_3_2_47_2","first-page":"30569","article-title":"DualCoOp: Fast adaptation to multi-label recognition with limited annotations","volume":"35","author":"Sun Ximeng","year":"2022","unstructured":"Ximeng Sun, Ping Hu, and Kate Saenko. 2022. DualCoOp: Fast adaptation to multi-label recognition with limited annotations. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 30569\u201330582.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_48_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.696"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.251"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2021.3083978"},{"key":"e_1_3_2_52_2","volume-title":"Proceedings of the 22nd International Joint Conference on Artificial Intelligence","author":"Weston Jason","year":"2011","unstructured":"Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. Wsabie: Scaling up to large vocabulary image annotation. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence. Citeseer."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00679"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2023.3268997"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2023.3288205"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.328"},{"key":"e_1_3_2_57_2","unstructured":"Shichao Xu Yikang Li Jenhao Hsiao Chiuman Ho and Zhu Qi. 2022. A dual modality approach for (zero-shot) multi-label classification. arXiv:2208.09562. Retrieved from https:\/\/arxiv.org\/abs\/2208.09562"},{"key":"e_1_3_2_58_2","first-page":"1909","article-title":"Dual-stream contrastive learning for compositional zero-shot recognition","author":"Yang Yanhua","year":"2023","unstructured":"Yanhua Yang, Rui Pan, Xiangyu Li, Xu Yang, and Cheng Deng. 2023. Dual-stream contrastive learning for compositional zero-shot recognition. IEEE Transactions on Multimedia 26 (2023), 1909\u20131919.","journal-title":"IEEE Transactions on Multimedia"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20077-9_7"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01416"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.321"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19833-5_29"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.644"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2021.3128054"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01631"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-022-01653-1"},{"key":"e_1_3_2_67_2","unstructured":"Xuelin Zhu Jiuxin Cao Dongqi Tang Furong Xu Weijia Liu Jiawei Ge Bo Liu Qingpei Guo and Tianyi Zhang. 2023. Text as image: Learning transferable adapter for multi-label classification. arXiv:2312.04160. Retrieved from https:\/\/arxiv.org\/abs\/2312.04160"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2024.3408256"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-96530-3"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3762195","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,22]],"date-time":"2025-11-22T07:00:20Z","timestamp":1763794820000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3762195"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,21]]},"references-count":68,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3762195"],"URL":"https:\/\/doi.org\/10.1145\/3762195","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,21]]},"assertion":[{"value":"2025-03-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}