{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T15:38:54Z","timestamp":1788622734849,"version":"build-2803163510"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"1s","license":[{"start":{"date-parts":[[2021,1,31]],"date-time":"2021-01-31T00:00:00Z","timestamp":1612051200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61976049 and 61632007"],"award-info":[{"award-number":["61976049 and 61632007"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","award":["ZYGX2019Z015"],"award-info":[{"award-number":["ZYGX2019Z015"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Sichuan Science and Technology Program, China","award":["2019ZDZX0008, 2019YFG0003, 2019YFG0533, 2019YFG0535, 2020YFS0057, and 2020YJ0038"],"award-info":[{"award-number":["2019ZDZX0008, 2019YFG0003, 2019YFG0533, 2019YFG0535, 2020YFS0057, and 2020YJ0038"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2021,1,31]]},"abstract":"<jats:p>Conventional cross-modal retrieval models mainly assume the same scope of the classes for both the training set and the testing set. This assumption limits their extensibility on zero-shot cross-modal retrieval (ZS-CMR), where the testing set consists of unseen classes that are disjoint with seen classes in the training set. The ZS-CMR task is more challenging due to the heterogeneous distributions of different modalities and the semantic inconsistency between seen and unseen classes. A few of recently proposed approaches are inspired by zero-shot learning to estimate the distribution underlying multimodal data by generative models and make the knowledge transfer from seen classes to unseen classes by leveraging class embeddings. However, directly borrowing the idea from zero-shot learning (ZSL) is not fully adaptive to the retrieval task, since the core of the retrieval task is learning the common space. To address the above issues, we propose a novel approach named Assembling AutoEncoder and Generative Adversarial Network (AAEGAN), which combines the strength of AutoEncoder (AE) and Generative Adversarial Network (GAN), to jointly incorporate common latent space learning, knowledge transfer, and feature synthesis for ZS-CMR. Besides, instead of utilizing class embeddings as common space, the AAEGAN approach maps all multimodal data into a learned latent space with the distribution alignment via three coupled AEs. We empirically show the remarkable improvement for ZS-CMR task and establish the state-of-the-art or competitive performance on four image-text retrieval datasets.<\/jats:p>","DOI":"10.1145\/3424341","type":"journal-article","created":{"date-parts":[[2021,4,1]],"date-time":"2021-04-01T01:53:55Z","timestamp":1617242035000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["Zero-shot Cross-modal Retrieval by Assembling AutoEncoder and Generative Adversarial Network"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5685-3123","authenticated-orcid":false,"given":"Xing","family":"Xu","sequence":"first","affiliation":[{"name":"University of Electronic Science and Technology of China, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jialin","family":"Tian","sequence":"additional","affiliation":[{"name":"University of Electronic Science and Technology of China, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaiyi","family":"Lin","sequence":"additional","affiliation":[{"name":"University of Electronic Science and Technology of China, Chengdu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huimin","family":"Lu","sequence":"additional","affiliation":[{"name":"Kyushu Institute of Technology, Kitakyushu, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2615-1555","authenticated-orcid":false,"given":"Jie","family":"Shao","sequence":"additional","affiliation":[{"name":"University of Electronic Science and Technology of China, China and Sichuan Artificial Intelligence Research Institute, Yibin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heng Tao","family":"Shen","sequence":"additional","affiliation":[{"name":"University of Electronic Science and Technology of China, China and Sichuan Artificial Intelligence Research Institute, Yibin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,3,31]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.14"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298911"},{"key":"e_1_2_1_3_1","unstructured":"Martin Arjovsky and L\u00e9on Bottou. 2017. Towards principled methods for training generative adversarial networks. arXiv:1701.04862. Retrieved from https:\/\/arxiv.org\/abs\/1701.04862.  Martin Arjovsky and L\u00e9on Bottou. 2017. Towards principled methods for training generative adversarial networks. arXiv:1701.04862. Retrieved from https:\/\/arxiv.org\/abs\/1701.04862."},{"key":"e_1_2_1_4_1","unstructured":"Martin Arjovsky Soumith Chintala and L\u00e9on Bottou. 2017. Wasserstein gan. arXiv:1701.07875. Retrieved from https:\/\/arxiv.org\/abs\/1701.07875.  Martin Arjovsky Soumith Chintala and L\u00e9on Bottou. 2017. Wasserstein gan. arXiv:1701.07875. Retrieved from https:\/\/arxiv.org\/abs\/1701.07875."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2578726.2578728"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00115"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/3304415.3304510"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2019.2900171"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1646396.1646452"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. 215\u2013223","author":"Coates Adam","year":"2011","unstructured":"Adam Coates , Andrew Ng , and Honglak Lee . 2011 . An analysis of single-layer networks in unsupervised feature learning . In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. 215\u2013223 . Adam Coates, Andrew Ng, and Honglak Lee. 2011. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. 215\u2013223."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654902"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-013-0658-4"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295327"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1162\/0899766042321814"},{"key":"e_1_2_1_15_1","volume-title":"Salakhutdinov","author":"Hinton Geoffrey E.","year":"2006","unstructured":"Geoffrey E. Hinton and Ruslan R . Salakhutdinov . 2006 . Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504\u2013507. Geoffrey E. Hinton and Ruslan R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504\u2013507."},{"key":"e_1_2_1_16_1","first-page":"143","article-title":"MHTN: Modal-adversarial hybrid transfer network for cross-modal retrieval","volume":"14","author":"Huang Xin","year":"2018","unstructured":"Xin Huang , Yuxin Peng , and Mingkuan Yuan . 2018 . MHTN: Modal-adversarial hybrid transfer network for cross-modal retrieval . IEEE Trans. Cybernet. 14 , 6 (2018), 143 \u2013 156 . Xin Huang, Yuxin Peng, and Mingkuan Yuan. 2018. MHTN: Modal-adversarial hybrid transfer network for cross-modal retrieval. IEEE Trans. Cybernet. 14, 6 (2018), 143\u2013156.","journal-title":"IEEE Trans. Cybernet."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2015.2390499"},{"key":"e_1_2_1_18_1","volume-title":"Kingma and Max Welling","author":"Diederik","year":"2013","unstructured":"Diederik P. Kingma and Max Welling . 2013 . Auto-encoding variational bayes. arXiv:1312.6114. Retrieved from https:\/\/arxiv.org\/abs\/1312.6114. Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv:1312.6114. Retrieved from https:\/\/arxiv.org\/abs\/1312.6114."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206594"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.140"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045390.3045555"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/3044805.3045025"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/957013.957143"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/957013.957143"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6817"},{"key":"e_1_2_1_26_1","unstructured":"Ruoyu Liu Yao Zhao Liang Zheng Shikui Wei and Yi Yang. 2017. A new evaluation protocol and benchmarking results for extendable cross-media retrieval. arXiv:1703.03567. Retrieved from https:\/\/arxiv.org\/abs\/1703.03567.  Ruoyu Liu Yao Zhao Liang Zheng Shikui Wei and Yi Yang. 2017. A new evaluation protocol and benchmarking results for extendable cross-media retrieval. arXiv:1703.03567. Retrieved from https:\/\/arxiv.org\/abs\/1703.03567."},{"key":"e_1_2_1_27_1","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781. Retrieved from https:\/\/arxiv.org\/abs\/1301.3781.  Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781. Retrieved from https:\/\/arxiv.org\/abs\/1301.3781."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/3061053.3061157"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3284750"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2017.2742704"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.466"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866717"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1873951.1873987"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/3045118.3045347"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00227"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2020.2970050"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-23528-8_9"},{"key":"e_1_2_1_38_1","unstructured":"K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556.  K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https:\/\/arxiv.org\/abs\/1409.1556."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/2999611.2999716"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the International Conference on Machine Learning Workshop.","author":"Srivastava N.","unstructured":"N. Srivastava and R. Salakhutdinov . 2012. Learning representations for multimodal data with deep belief nets . In Proceedings of the International Conference on Machine Learning Workshop. N. Srivastava and R. Salakhutdinov. 2012. Learning representations for multimodal data with deep belief nets. In Proceedings of the International Conference on Machine Learning Workshop."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/2999325.2999383"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1199\u20131208","author":"Sung Flood","unstructured":"Flood Sung , Yongxin Yang , Li Zhang , Tao Xiang , Philip H. S. Torr , and Timothy M. Hospedales . 2018. Learning to compare: Relation network for few-shot learning . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1199\u20131208 . Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1199\u20131208."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390294"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123266.3123326"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2505311"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.261"},{"key":"e_1_2_1_47_1","unstructured":"Yang Wang. 2020. Survey on deep multi-modal data analytics: Collaboration rivalry and fusion. arXiv:2006.08159. Retrieved from https:\/\/arxiv.org\/abs\/2006.08159.  Yang Wang. 2020. Survey on deep multi-modal data analytics: Collaboration rivalry and fusion. arXiv:2006.08159. Retrieved from https:\/\/arxiv.org\/abs\/2006.08159."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2655449"},{"key":"e_1_2_1_49_1","first-page":"449","article-title":"Cross-modal retrieval with CNN visual features: A new baseline","volume":"47","author":"Wei Yunchao","year":"2017","unstructured":"Yunchao Wei , Yao Zhao , Canyi Lu , Shikui Wei , Luoqi Liu , Zhenfeng Zhu , and Shuicheng Yan . 2017 . Cross-modal retrieval with CNN visual features: A new baseline . IEEE Trans. Cybernet. 47 , 2 (2017), 449 \u2013 460 . Yunchao Wei, Yao Zhao, Canyi Lu, Shikui Wei, Luoqi Liu, Zhenfeng Zhu, and Shuicheng Yan. 2017. Cross-modal retrieval with CNN visual features: A new baseline. IEEE Trans. Cybernet. 47, 2 (2017), 449\u2013460.","journal-title":"IEEE Trans. Cybernet."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2018.2877886"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2878970"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00581"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01052"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-018-0541-x"},{"key":"e_1_2_1_55_1","volume-title":"Heng Tao Shen, and Xuelong Li","author":"Xu Xing","year":"2020","unstructured":"Xing Xu , Kaiyi Lin , Lianli Gao , Huimin Lu , Heng Tao Shen, and Xuelong Li . 2020 . Cross-modal common representations by private-shared subspaces separation. IEEE Trans. Cybernet . (2020), 1\u201314. https:\/\/ieeexplore.ieee.org\/document\/9165187. Xing Xu, Kaiyi Lin, Lianli Gao, Huimin Lu, Heng Tao Shen, and Xuelong Li. 2020. Cross-modal common representations by private-shared subspaces separation. IEEE Trans. Cybernet. (2020), 1\u201314. https:\/\/ieeexplore.ieee.org\/document\/9165187."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2019.2928180"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3206025.3206033"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR\u201915)","author":"Yan F.","unstructured":"F. Yan and K. Mikolajczyk . 2015. Deep correlation for matching images and text . In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR\u201915) . 3441\u20133450. F. Yan and K. Mikolajczyk. 2015. Deep correlation for matching images and text. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR\u201915). 3441\u20133450."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2013.2276704"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.321"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00111"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.5555\/2891460.2891609"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3424341","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3424341","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:26Z","timestamp":1750197746000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3424341"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,31]]},"references-count":62,"journal-issue":{"issue":"1s","published-print":{"date-parts":[[2021,1,31]]}},"alternative-id":["10.1145\/3424341"],"URL":"https:\/\/doi.org\/10.1145\/3424341","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,1,31]]},"assertion":[{"value":"2020-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-03-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}