{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T09:46:19Z","timestamp":1773913579355,"version":"3.50.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,9,18]],"date-time":"2023-09-18T00:00:00Z","timestamp":1694995200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62276266, 61801198, 62272461, 62172417, 62106268"],"award-info":[{"award-number":["62276266, 61801198, 62272461, 62172417, 62106268"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>Diverse image captioning has achieved substantial progress in recent years. However, the discriminability of generative models and the limitation of cross entropy loss are generally overlooked in the traditional diverse image captioning models, which seriously hurts both the diversity and accuracy of image captioning. In this article, aiming to improve diversity and accuracy simultaneously, we propose a novel Conditional Variational Autoencoder (DCL-CVAE) framework for diverse image captioning by seamlessly integrating sequential variational autoencoder with contrastive learning. In the encoding stage, we first build conditional variational autoencoders to separately learn the sequential latent spaces for a pair of captions. Then, we introduce contrastive learning in the sequential latent spaces to enhance the discriminability of latent representations for both image-caption pairs and mismatched pairs. In the decoding stage, we leverage the captions sampled from the pre-trained Long Short-Term Memory (LSTM), LSTM decoder as the negative examples and perform contrastive learning with the greedily sampled positive examples, which can restrain the generation of common words and phrases induced by the cross entropy loss. By virtue of dual constrastive learning, DCL-CVAE is capable of encouraging the discriminability and facilitating the diversity, while promoting the accuracy of the generated captions. Extensive experiments are conducted on the challenging MSCOCO dataset, showing that our proposed methods can achieve a better balance between accuracy and diversity compared to the state-of-the-art diverse image captioning models.<\/jats:p>","DOI":"10.1145\/3614435","type":"journal-article","created":{"date-parts":[[2023,8,11]],"date-time":"2023-08-11T11:16:06Z","timestamp":1691752566000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Diverse Image Captioning via Conditional Variational Autoencoder and Dual Contrastive Learning"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2980-1755","authenticated-orcid":false,"given":"Jing","family":"Xu","sequence":"first","affiliation":[{"name":"School of Computer Science and Technology, China University of Mining and Technology, Engineering Research Center of Mine Digitization, Ministry of Education of the Peoples Republic of China, Xuzhou, Jiangsu, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2365-6606","authenticated-orcid":false,"given":"Bing","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, China University of Mining and Technology, Engineering Research Center of Mine Digitization, Ministry of Education of the Peoples Republic of China, Xuzhou, Jiangsu, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6207-0299","authenticated-orcid":false,"given":"Yong","family":"Zhou","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, China University of Mining and Technology, Engineering Research Center of Mine Digitization, Ministry of Education of the Peoples Republic of China, Xuzhou, Jiangsu, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5698-8308","authenticated-orcid":false,"given":"Mingming","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, China University of Mining and Technology, Engineering Research Center of Mine Digitization, Ministry of Education of the Peoples Republic of China, Xuzhou, Jiangsu, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2734-915X","authenticated-orcid":false,"given":"Rui","family":"Yao","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, China University of Mining and Technology, Engineering Research Center of Mine Digitization, Ministry of Education of the Peoples Republic of China, Xuzhou, Jiangsu, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9383-8384","authenticated-orcid":false,"given":"Zhiwen","family":"Shao","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, China University of Mining and Technology, Engineering Research Center of Mine Digitization, Ministry of Education of the Peoples Republic of China, Xuzhou, Jiangsu, China"}]}],"member":"320","published-online":{"date-parts":[[2023,9,18]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"6697","volume-title":"Proceedings of the 31st Conference on Neural Information Processing Systems","author":"Goyal Anirudh Goyal Alias Parth","year":"2017","unstructured":"Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre C\u00f4t\u00e9, Nan Rosemary Ke, and Yoshua Bengio. 2017. Z-forcing: Training stochastic recurrent networks. In Proceedings of the 31st Conference on Neural Information Processing Systems. 6697\u20136707."},{"key":"e_1_3_2_3_2","first-page":"382","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201916)","author":"Anderson Peter","year":"2016","unstructured":"Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Proceedings of the European Conference on Computer Vision (ECCV\u201916). Springer, 382\u2013398."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00636"},{"key":"e_1_3_2_5_2","first-page":"4261","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV\u201919)","author":"Aneja Jyoti","year":"2019","unstructured":"Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. 2019. Sequential latent spaces for modeling the intention during diverse image captioning. In Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV\u201919). 4261\u20134270."},{"key":"e_1_3_2_6_2","first-page":"2422","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915)","author":"Chen Xinlei","year":"2015","unstructured":"Xinlei Chen and C. Lawrence Zitnick. 2015. Mind\u2019s eye: A recurrent visual representation for image caption generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915). 2422\u20132431."},{"key":"e_1_3_2_7_2","first-page":"2980","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS\u201915)","author":"Chung Junyoung","year":"2015","unstructured":"Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS\u201915). 2980\u20132988."},{"key":"e_1_3_2_8_2","first-page":"10578","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201920)","author":"Cornia Marcella","year":"2020","unstructured":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201920). 10578\u201310587."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.5555\/3294771.3294857"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3348"},{"key":"e_1_3_2_11_2","first-page":"10695","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201919)","author":"Deshpande Aditya","year":"2019","unstructured":"Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander G. Schwing, and David Forsyth. 2019. Fast, diverse and accurate image captioning guided by part-of-speech. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201919). 10695\u201310704."},{"key":"e_1_3_2_12_2","unstructured":"Jacob Devlin Saurabh Gupta Ross Girshick Margaret Mitchell and C. Lawrence Zitnick. 2015. Exploring nearest neighbor approaches for image captioning. CoRRabs\/1505.04467 2015."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"e_1_3_2_14_2","first-page":"15","volume-title":"Proceedings of the 11th European Conference on Computer Vision (ECCV\u201910)","author":"Farhadi Ali","year":"2010","unstructured":"Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In Proceedings of the 11th European Conference on Computer Vision (ECCV\u201910). Springer, 15\u201329."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/3157096.3157343"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-3207"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_18_2","first-page":"4634","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV\u201919)","author":"Huang Lun","year":"2019","unstructured":"Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019. Attention on attention for image captioning. In Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV\u201919). 4634\u20134643."},{"key":"e_1_3_2_19_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR\u201917)","author":"Jang Eric","year":"2017","unstructured":"Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In Proceedings of the International Conference on Learning Representations (ICLR\u201917)."},{"key":"e_1_3_2_20_2","first-page":"2407","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201915)","author":"Jia Xu","year":"2015","unstructured":"Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. 2015. Guiding the long-short term memory model for image caption generation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201915). 2407\u20132415."},{"key":"e_1_3_2_21_2","first-page":"3128","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915)","author":"Karpathy Andrej","year":"2015","unstructured":"Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915). 3128\u20133137."},{"key":"e_1_3_2_22_2","first-page":"8887","volume-title":"Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV\u201919)","author":"Ke Lei","year":"2019","unstructured":"Lei Ke, Wenjie Pei, Ruiyu Li, Xiaoyong Shen, and Yu-Wing Tai. 2019. Reflective decoding network for image captioning. In Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV\u201919). 8887\u20138896."},{"key":"e_1_3_2_23_2","first-page":"317","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915)","author":"Krause Jonathan","year":"2017","unstructured":"Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generating descriptive image paragraphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915). 317\u2013325."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/tpami.2012.162"},{"key":"e_1_3_2_25_2","volume-title":"Proceedings of the Conference on Neural Information Processing Systems (NeurIPS\u201918)","author":"Li Dianqi","year":"2018","unstructured":"Dianqi Li, Qiuyuan Huang, Xiaodong He, Lei Zhang, and Ming-Ting Sun. 2018. Generating diverse and accurate visual captions by comparative adversarial learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS\u201918)."},{"key":"e_1_3_2_26_2","first-page":"74","volume-title":"Proceedings of the Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Text Summarization Branches Out. 74\u201381."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_28_2","first-page":"338","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201918)","author":"Liu Xihui","year":"2018","unstructured":"Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. 2018. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In Proceedings of the European Conference on Computer Vision (ECCV\u201918). Springer, 338\u2013354."},{"key":"e_1_3_2_29_2","volume-title":"Proceedings of the 8th International Conference on Learning Representations (ICLR\u201920)","author":"Mahajan Shweta","year":"2020","unstructured":"Shweta Mahajan, Iryna Gurevych, and Stefan Roth. 2020. Latent normalizing flows for many-to-many cross-domain mappings. In Proceedings of the 8th International Conference on Learning Representations (ICLR\u201920)."},{"key":"e_1_3_2_30_2","first-page":"3613","article-title":"Diverse image captioning with context-object split latent spaces","volume":"33","author":"Mahajan Shweta","year":"2020","unstructured":"Shweta Mahajan and Stefan Roth. 2020. Diverse image captioning with context-object split latent spaces. Advances in Neural Information Processing Systems 33 (2020), 3613\u20133624.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_31_2","article-title":"Deep captioning with multimodal recurrent neural networks (m-RNN)","author":"Mao Junhua","year":"2015","unstructured":"Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2015. Deep captioning with multimodal recurrent neural networks (m-RNN). In Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915).","journal-title":"Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915)"},{"key":"e_1_3_2_32_2","first-page":"10971","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201920)","author":"Pan Yingwei","year":"2020","unstructured":"Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020. X-linear attention networks for image captioning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR\u201920). 10971\u201310980."},{"key":"e_1_3_2_33_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2019.00034"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2016.2577031"},{"key":"e_1_3_2_36_2","first-page":"290","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917)","author":"Ren Zhou","year":"2017","unstructured":"Zhou Ren, Xiaoyu Wang, Ning Zhang, Xutao Lv, and Li-Jia Li. 2017. Deep reinforcement learning-based image captioning with embedding reward. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917). 290\u2013298."},{"key":"e_1_3_2_37_2","first-page":"4135","volume-title":"Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917)","author":"Shetty Rakshith","year":"2017","unstructured":"Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. 2017. Speaking the same language: Matching machine to human captions by adversarial training. In Proceedings of the IEEE International Conference on Computer Vision (ICCV\u201917). 4135\u20134144."},{"issue":"4","key":"e_1_3_2_38_2","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1109\/TSMC.1971.4308316","article-title":"A stochastic approximation method","author":"Sinha Naresh K.","year":"1971","unstructured":"Naresh K. Sinha and Michael P. Griscik. 1971. A stochastic approximation method. IEEE Transactions on Systems, Man, and Cybernetics4 (1971), 338\u2013344.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"e_1_3_2_39_2","first-page":"1107","article-title":"Matching words and pictures","volume":"3","author":"Jordan Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M. Blei, and Michael I.","year":"2003","unstructured":"Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M. Blei, and Michael I. Jordan. 2003. Matching words and pictures. The Journal of Machine Learning Research 3, 38 (2003), 1107\u20131135.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_3_2_40_2","first-page":"7371","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI\u201918)","author":"Batra Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv","year":"2018","unstructured":"Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI\u201918). 7371\u20137379."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"e_1_3_2_42_2","first-page":"8898","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision (CVPR\u201919)","author":"Vered Gilad","year":"2020","unstructured":"Gilad Vered, Gal Oren, Yuval Atzmon, and Gal Chechik. 2020. Joint optimization for cooperative image captioning. In Proceedings of the IEEE\/CVF International Conference on Computer Vision (CVPR\u201919). 8898\u20138907."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"e_1_3_2_44_2","first-page":"1","article-title":"On distinctive image captioning via comparing and reweighting","author":"Wang Jiuniu","year":"2022","unstructured":"Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B. Chan. 2022. On distinctive image captioning via comparing and reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022), 1\u20131.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295326"},{"key":"e_1_3_2_46_2","article-title":"Review networks for caption generation","author":"Yang Zhilin","year":"2016","unstructured":"Zhilin Yang, Ye Yuan, Yuexin Wu, William W. Cohen, and Russ R. Salakhutdinov. 2016. Review networks for caption generation. Advances in Neural Information Processing Systems 29 (2016), 2361\u20132369.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_47_2","first-page":"711","volume-title":"Proceedings of the 15th European Conference on Computer Vision (ECCV\u201918)","volume":"11218","author":"Yao Ting","year":"2018","unstructured":"Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring visual relationship for image captioning. In Proceedings of the 15th European Conference on Computer Vision (ECCV\u201918), Vol. 11218. 711\u2013727."},{"key":"e_1_3_2_48_2","first-page":"4651","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201916)","author":"You Quanzeng","year":"2016","unstructured":"Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201916). 4651\u20134659."}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3614435","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3614435","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:30Z","timestamp":1750287030000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3614435"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,18]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3614435"],"URL":"https:\/\/doi.org\/10.1145\/3614435","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,18]]},"assertion":[{"value":"2022-12-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-28","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}