{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,5]],"date-time":"2026-08-05T10:08:53Z","timestamp":1785924533581,"version":"3.56.0"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,6,6]],"date-time":"2025-06-06T00:00:00Z","timestamp":1749168000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>\n            In natural language generation, language models, particularly those based on decoder-only architectures as in popular Large Language Models (LLMs), have demonstrated impressive performance across a wide range of tasks. However, encoder-decoder architectures remain highly effective for tasks involving non-text data, such as images and time-series data. The decoder relies on the attention mechanism to efficiently extract information from the encoder. While it is common practice to draw information from only the last encoder layer, this might lead to insufficient training of the encoder layer stack due to the\n            <jats:italic>hierarchy bypassing<\/jats:italic>\n            problem. In this work, we propose\n            <jats:italic>layer-wise multi-view decoding<\/jats:italic>\n            for improved encoder-decoder language models, where for each decoder layer, together with the representations from the last encoder layer, which serve as a global view, those from other encoder layers are supplemented for a stereoscopic view of the source inputs. Systematic experiments and analyses show that we successfully address the hierarchy bypassing problem, require almost negligible parameter increase, and improve the performance of sequence learning with deep representations on diverse tasks, i.e., machine translation, abstractive summarization, image captioning, video captioning, medical report generation, and paraphrase generation. In particular, our approach achieves new state-of-the-art results on benchmark datasets, including a low-resource machine translation dataset and low-resource medical report generation datasets.\n          <\/jats:p>","DOI":"10.1145\/3729536","type":"journal-article","created":{"date-parts":[[2025,4,15]],"date-time":"2025-04-15T16:25:46Z","timestamp":1744734346000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Rethinking Natural Language Generation with Layer-Wise Multi-View Decoding"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7715-5228","authenticated-orcid":false,"given":"Fenglin","family":"Liu","sequence":"first","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6994-2114","authenticated-orcid":false,"given":"Xuancheng","family":"Ren","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3046-512X","authenticated-orcid":false,"given":"Guangxiang","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8365-7822","authenticated-orcid":false,"given":"Chenyu","family":"You","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Yale University, New Haven, Connecticut, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5639-591X","authenticated-orcid":false,"given":"Sherry","family":"Ma","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1118-9710","authenticated-orcid":false,"given":"Xian","family":"Wu","sequence":"additional","affiliation":[{"name":"Tencent Youtu Lab, Jarvis Research Center, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8363-3974","authenticated-orcid":false,"given":"Wei","family":"Fan","sequence":"additional","affiliation":[{"name":"Tencent America LLC, Palo Alto, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8241-9320","authenticated-orcid":false,"given":"Xu","family":"Sun","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,6]]},"reference":[{"key":"e_1_3_3_2_2","article-title":"Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning","author":"Aafaq Nayyer","year":"2019","unstructured":"Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. 2019. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_3_2","article-title":"SPICE: Semantic propositional image caption evaluation","author":"Anderson Peter","year":"2016","unstructured":"Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: Semantic propositional image caption evaluation. In ECCV.","journal-title":"ECCV"},{"key":"e_1_3_3_4_2","article-title":"Bottom-up and top-down attention for image captioning and VQA","author":"Anderson Peter","year":"2018","unstructured":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and VQA. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_5_2","unstructured":"Lei Jimmy Ba Ryan Kiros and Geoffrey E. Hinton. 2016. Layer normalization. arXiv:1607.06450. Retrieved from https:\/\/arxiv.org\/abs\/1607.06450"},{"key":"e_1_3_3_6_2","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau Dzmitry","year":"2015","unstructured":"Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.","journal-title":"ICLR"},{"key":"e_1_3_3_7_2","volume-title":"ACL Workshop","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop."},{"key":"e_1_3_3_8_2","article-title":"Training deeper neural machine translation models with transparent attention","author":"Bapna Ankur","year":"2018","unstructured":"Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In EMNLP.","journal-title":"EMNLP"},{"key":"e_1_3_3_9_2","unstructured":"Xinlei Chen Hao Fang Tsung-Yi Lin Ramakrishna Vedantam Saurabh Gupta Piotr Doll\u00e1r and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv:1504.00325. Retrieved from https:\/\/arxiv.org\/abs\/1504.00325"},{"key":"e_1_3_3_10_2","article-title":"Cross-modal memory networks for radiology report generation","author":"Chen Zhihong","year":"2021","unstructured":"Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021. Cross-modal memory networks for radiology report generation. In ACL\/IJCNLP.","journal-title":"ACL\/IJCNLP"},{"key":"e_1_3_3_11_2","article-title":"Generating radiology reports via memory-driven transformer","author":"Chen Zhihong","year":"2020","unstructured":"Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. 2020. Generating radiology reports via memory-driven transformer. In EMNLP.","journal-title":"EMNLP"},{"key":"e_1_3_3_12_2","article-title":"Meshed-memory transformer for image captioning","author":"Cornia Marcella","year":"2020","unstructured":"Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In CVPR.","journal-title":"CVPR"},{"issue":"2","key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1093\/jamia\/ocv080","article-title":"Preparing a collection of radiology examinations for distribution and retrieval","volume":"23","author":"Demner-Fushman Dina","year":"2016","unstructured":"Dina Demner-Fushman, Marc D. Kohli, Marc B. Rosenman, Sonya E. Shooshan, Laritza Rodriguez, Sameer K. Antani, George R. Thoma, and Clement J. McDonald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association 23, 2 (2016), 304\u2013310.","journal-title":"Journal of the American Medical Informatics Association"},{"key":"e_1_3_3_14_2","article-title":"ImageNet: A large-scale hierarchical image database","author":"Deng Jia","year":"2009","unstructured":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_15_2","article-title":"How much attention do you need? A granular analysis of neural machine translation architectures","author":"Domhan Tobias","year":"2018","unstructured":"Tobias Domhan. 2018. How much attention do you need? A granular analysis of neural machine translation architectures. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_16_2","first-page":"875","article-title":"Learning to paraphrase for question answering","author":"Dong Li","year":"2017","unstructured":"Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In EMNLP, 875\u2013886.","journal-title":"EMNLP"},{"key":"e_1_3_3_17_2","article-title":"Exploiting deep representations for neural machine translation","author":"Dou Zi-Yi","year":"2018","unstructured":"Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Shuming Shi, and Tong Zhang. 2018. Exploiting deep representations for neural machine translation. In EMNLP.","journal-title":"EMNLP"},{"key":"e_1_3_3_18_2","article-title":"Dynamic layer aggregation for neural machine translation with routing-by-agreement","author":"Dou Zi-Yi","year":"2019","unstructured":"Zi-Yi Dou, Zhaopeng Tu, Xing Wang, Longyue Wang, Shuming Shi, and Tong Zhang. 2019. Dynamic layer aggregation for neural machine translation with routing-by-agreement. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_19_2","article-title":"Understanding back-translation at scale","author":"Edunov Sergey","year":"2018","unstructured":"Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In EMNLP.","journal-title":"EMNLP"},{"key":"e_1_3_3_20_2","first-page":"45","volume-title":"NMT@ACL","author":"Fan Angela","year":"2018","unstructured":"Angela Fan, David Grangier, and Michael Auli. 2018. Controllable abstractive summarization. In NMT@ACL. Association for Computational Linguistics, 45\u201354."},{"key":"e_1_3_3_21_2","article-title":"YouTube2Text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition","author":"Guadarrama Sergio","year":"2013","unstructured":"Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. 2013. YouTube2Text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV.","journal-title":"ICCV"},{"key":"e_1_3_3_22_2","article-title":"Deep residual learning for image recognition","author":"He Kaiming","year":"2016","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_23_2","article-title":"Layer-wise coordination between encoder and decoder for neural machine translation","author":"He Tianyu","year":"2018","unstructured":"Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Layer-wise coordination between encoder and decoder for neural machine translation. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_3_24_2","article-title":"Image captioning: Transforming objects into words","author":"Herdade Simao","year":"2019","unstructured":"Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019. Image captioning: Transforming objects into words. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_3_25_2","article-title":"Teaching machines to read and comprehend","author":"Moritz Hermann Karl","year":"2015","unstructured":"Karl Moritz Hermann, Tom\u00e1s Kocisk\u00fd, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS.","journal-title":"NIPS"},{"key":"e_1_3_3_26_2","article-title":"CNN architectures for large-scale audio classification","author":"Hershey Shawn","year":"2017","unstructured":"Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In ICASSP.","journal-title":"ICASSP"},{"key":"e_1_3_3_27_2","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v33i01.3301590","article-title":"CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison","author":"Irvin Jeremy","year":"2019","unstructured":"Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn L. Ball, Katie S. Shpanskaya, et al. 2019. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_28_2","first-page":"1875","article-title":"Adversarial example generation with syntactically controlled paraphrase networks","author":"Iyyer Mohit","year":"2018","unstructured":"Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In NAACL-HLT, 1875\u20131885.","journal-title":"NAACL-HLT"},{"key":"e_1_3_3_29_2","volume-title":"NeurIPS (Workshop)","author":"Jeffrey Niall","year":"2020","unstructured":"Niall Jeffrey and Benjamin D. Wandelt. 2020. Solving high-dimensional parameter inference: Marginal posterior densities & moment networks. In NeurIPS (Workshop)."},{"key":"e_1_3_3_30_2","article-title":"Recurrent fusion network for image captioning","author":"Jiang Wenhao","year":"2018","unstructured":"Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. 2018. Recurrent fusion network for image captioning. In ECCV.","journal-title":"ECCV"},{"key":"e_1_3_3_31_2","article-title":"Show, describe and conclude: On exploiting the structure information of chest X-ray reports","author":"Jing Baoyu","year":"2019","unstructured":"Baoyu Jing, Zeya Wang, and Eric P. Xing. 2019. Show, describe and conclude: On exploiting the structure information of chest X-ray reports. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_32_2","article-title":"On the automatic generation of medical imaging reports","author":"Jing Baoyu","year":"2018","unstructured":"Baoyu Jing, Pengtao Xie, and Eric P. Xing. 2018. On the automatic generation of medical imaging reports. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","unstructured":"Alistair E. W. Johnson Tom J. Pollard Seth J. Berkowitz Nathaniel R. Greenbaum Matthew P. Lungren Chih-Ying Deng Roger G. Mark and Steven Horng. 2019. MIMIC-CXR: A large publicly available database of labeled chest radiographs. arXiv:1901.07042. Retrieved from https:\/\/arxiv.org\/abs\/1901.07042","DOI":"10.1038\/s41597-019-0322-0"},{"key":"e_1_3_3_34_2","article-title":"Deep visual-semantic alignments for generating image descriptions","author":"Karpathy Andrej","year":"2015","unstructured":"Andrej Karpathy and Fei-Fei Li. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_35_2","article-title":"Large-scale video classification with convolutional neural networks","author":"Karpathy Andrej","year":"2014","unstructured":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Fei-Fei Li. 2014. Large-scale video classification with convolutional neural networks. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_36_2","article-title":"OpenNMT: Open-source toolkit for neural machine translation","author":"Klein Guillaume","year":"2017","unstructured":"Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_37_2","first-page":"1224","article-title":"A continuously growing dataset of sentential paraphrases","author":"Lan Wuwei","year":"2017","unstructured":"Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In EMNLP, 1224\u20131234.","journal-title":"EMNLP"},{"issue":"4","key":"e_1_3_3_38_2","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1162\/neco.1989.1.4.541","article-title":"Backpropagation applied to handwritten zip code recognition","volume":"1","author":"LeCun Yann","year":"1989","unstructured":"Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural Computation 1, 4 (1989), 541\u2013551.","journal-title":"Neural Computation"},{"key":"e_1_3_3_39_2","article-title":"Pre-training via paraphrasing","author":"Lewis Mike","year":"2020","unstructured":"Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_3_40_2","doi-asserted-by":"crossref","unstructured":"Bei Li Yi Jing Xu Tan Zhen Xing Tong Xiao and Jingbo Zhu. 2023. TranSFormer: Slow-fast transformer for machine translation. arXiv:2305.16982. Retrieved from https:\/\/arxiv.org\/abs\/2305.16982","DOI":"10.18653\/v1\/2023.findings-acl.430"},{"key":"e_1_3_3_41_2","first-page":"13225","volume-title":"ICML","author":"Li Bei","year":"2022","unstructured":"Bei Li, Tong Zheng, Yi Jing, Chengbo Jiao, Tong Xiao, and Jingbo Zhu. 2022. Learning multiscale transformer models for sequence generation. In ICML. PMLR, 13225\u201313241."},{"key":"e_1_3_3_42_2","article-title":"Knowledge-driven encode, retrieve, paraphrase for medical image report generation","author":"Li Christy Y.","year":"2019","unstructured":"Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2019. Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_43_2","first-page":"8927","volume-title":"ICCV","author":"Li Guang","year":"2019","unstructured":"Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. 2019. Entangled transformer for image captioning. In ICCV. IEEE, 8927\u20138936."},{"key":"e_1_3_3_44_2","article-title":"Neuron interaction based representation composition for neural machine translation","author":"Li Jian","year":"2020","unstructured":"Jian Li, Xing Wang, Baosong Yang, Shuming Shi, Michael R. Lyu, and Zhaopeng Tu. 2020. Neuron interaction based representation composition for neural machine translation. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_45_2","article-title":"Hybrid retrieval-generation reinforced agent for medical image report generation","author":"Li Yuan","year":"2018","unstructured":"Yuan Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2018. Hybrid retrieval-generation reinforced agent for medical image report generation. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_3_46_2","first-page":"3865","article-title":"Paraphrase generation with deep reinforcement learning","author":"Li Zichao","year":"2018","unstructured":"Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase generation with deep reinforcement learning. In EMNLP, 3865\u20133878.","journal-title":"EMNLP"},{"key":"e_1_3_3_47_2","article-title":"ROUGE: A package for automatic evaluation of summaries","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_48_2","article-title":"Context-aware visual policy network for sequence-level image captioning","author":"Liu Daqing","year":"2018","unstructured":"Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, and Feng Wu. 2018. Context-aware visual policy network for sequence-level image captioning. In ACM\/ICM.","journal-title":"ACM\/ICM"},{"key":"e_1_3_3_49_2","article-title":"Exploring and distilling cross-modal information for image captioning","author":"Liu Fenglin","year":"2019","unstructured":"Fenglin Liu, Xuancheng Ren, Yuanxin Liu, Kai Lei, and Xu Sun. 2019. Exploring and distilling cross-modal information for image captioning. In IJCAI.","journal-title":"IJCAI"},{"key":"e_1_3_3_50_2","article-title":"Prophet attention: Predicting attention with future attention","author":"Liu Fenglin","year":"2020","unstructured":"Fenglin Liu, Xuancheng Ren, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou, and Xu Sun. 2020. Prophet attention: Predicting attention with future attention. In NeurIPS.","journal-title":"NeurIPS"},{"key":"e_1_3_3_51_2","article-title":"Exploring and distilling posterior and prior knowledge for radiology report generation","author":"Liu Fenglin","year":"2021","unstructured":"Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2021. Exploring and distilling posterior and prior knowledge for radiology report generation. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_52_2","article-title":"Generating paraphrase with topic as prior knowledge","author":"Liu Yuanxin","year":"2019","unstructured":"Yuanxin Liu, Zheng Lin, Fenglin Liu, Qinyun Dai, and Weiping Wang. 2019. Generating paraphrase with topic as prior knowledge. In CIKM.","journal-title":"CIKM"},{"key":"e_1_3_3_53_2","article-title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","author":"Lu Jiasen","year":"2017","unstructured":"Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_54_2","article-title":"Attention calibration for transformer in neural machine translation","author":"Lu Yu","year":"2021","unstructured":"Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. 2021. Attention calibration for transformer in neural machine translation. In ACL\/IJCNLP.","journal-title":"ACL\/IJCNLP"},{"key":"e_1_3_3_55_2","article-title":"Effective approaches to attention-based neural machine translation","author":"Luong Thang","year":"2015","unstructured":"Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP.","journal-title":"EMNLP"},{"issue":"3","key":"e_1_3_3_56_2","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1162\/coli_a_00002","article-title":"Generating phrasal and sentential paraphrases: A survey of data-driven methods","volume":"36","author":"Madnani Nitin","year":"2010","unstructured":"Nitin Madnani and Bonnie J. Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics 36, 3 (2010), 341\u2013387.","journal-title":"Computational Linguistics"},{"key":"e_1_3_3_57_2","article-title":"Prevent the language model from being overconfident in neural machine translation","author":"Miao Mengqi","year":"2021","unstructured":"Mengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, and Jie Zhou. 2021. Prevent the language model from being overconfident in neural machine translation. In ACL\/IJCNLP.","journal-title":"ACL\/IJCNLP"},{"key":"e_1_3_3_58_2","article-title":"fairseq: A fast, extensible toolkit for sequence modeling","author":"Ott Myle","year":"2019","unstructured":"Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT.","journal-title":"NAACL-HLT"},{"key":"e_1_3_3_59_2","first-page":"1","article-title":"Scaling neural machine translation","author":"Ott Myle","year":"2018","unstructured":"Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In WMT, 1\u20139.","journal-title":"WMT"},{"key":"e_1_3_3_60_2","article-title":"Spatio-temporal graph for video captioning with knowledge distillation","author":"Pan Boxiao","year":"2020","unstructured":"Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. 2020. Spatio-temporal graph for video captioning with knowledge distillation. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_61_2","article-title":"X-linear attention networks for image captioning","author":"Pan Yingwei","year":"2020","unstructured":"Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020. X-linear attention networks for image captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_62_2","article-title":"Softening quantization in bag-of-audio-words","author":"Pancoast Stephanie","year":"2014","unstructured":"Stephanie Pancoast and Murat Akbacak. 2014. Softening quantization in bag-of-audio-words. In ICASSP.","journal-title":"ICASSP"},{"key":"e_1_3_3_63_2","article-title":"BLEU: A method for automatic evaluation of machine translation","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_64_2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1007\/s40593-016-0128-6","article-title":"Wise crowd content assessment and educational rubrics","volume":"28","author":"Passonneau Rebecca J.","year":"2018","unstructured":"Rebecca J. Passonneau, Ananya Poddar, Gaurav Gite, Alisa Krivokapic, Qian Yang, and Dolores Perin. 2018. Wise crowd content assessment and educational rubrics. International Journal of Artificial Intelligence in Education 28 (2018), 29\u201355.","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"e_1_3_3_65_2","article-title":"A deep reinforced model for abstractive summarization","author":"Paulus Romain","year":"2018","unstructured":"Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In ICLR.","journal-title":"ICLR"},{"key":"e_1_3_3_66_2","article-title":"Memory-attended recurrent network for video captioning","author":"Pei Wenjie","year":"2019","unstructured":"Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. 2019. Memory-attended recurrent network for video captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_67_2","article-title":"Deep contextualized word representations","author":"Peters Matthew E.","year":"2018","unstructured":"Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT.","journal-title":"NAACL-HLT"},{"key":"e_1_3_3_68_2","first-page":"2923","article-title":"Neural paraphrase generation with stacked residual LSTM networks","author":"Prakash Aaditya","year":"2018","unstructured":"Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek V. Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2018. Neural paraphrase generation with stacked residual LSTM networks. In COLING, 2923\u20132934.","journal-title":"COLING"},{"issue":"140","key":"e_1_3_3_69_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1\u201367.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_70_2","article-title":"Self-critical sequence training for image captioning","author":"Rennie Steven J.","year":"2017","unstructured":"Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_71_2","article-title":"Semantic grouping network for video captioning","author":"Ryu Hobin","year":"2021","unstructured":"Hobin Ryu, Sunghun Kang, Haeyong Kang, and Chang D. Yoo. 2021. Semantic grouping network for video captioning. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_72_2","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1007\/s11263-013-0636-x","article-title":"Image classification with the Fisher vector: Theory and practice","volume":"105","author":"S\u00e1nchez Jorge","year":"2013","unstructured":"Jorge S\u00e1nchez, Florent Perronnin, Thomas Mensink, and Jakob J. Verbeek. 2013. Image classification with the Fisher vector: Theory and practice. International Journal of Computer Vision, 105 (2013), 222\u2013245.","journal-title":"International Journal of Computer Vision"},{"key":"e_1_3_3_73_2","first-page":"1073","article-title":"Get to the point: Summarization with pointer-generator networks","author":"See Abigail","year":"2017","unstructured":"Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL, 1073\u20131083.","journal-title":"ACL"},{"key":"e_1_3_3_74_2","article-title":"Dense information flow for neural machine translation","author":"Shen Yanyao","year":"2018","unstructured":"Yanyao Shen, Xu Tan, Di He, Tao Qin, and Tie-Yan Liu. 2018. Dense information flow for neural machine translation. In NAACL-HLT.","journal-title":"NAACL-HLT"},{"key":"e_1_3_3_75_2","article-title":"The evolved transformer","author":"So David R.","year":"2019","unstructured":"David R. So, Quoc V. Le, and Chen Liang. 2019. The evolved transformer. In ICML.","journal-title":"ICML"},{"issue":"1","key":"e_1_3_3_76_2","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava Nitish","year":"2014","unstructured":"Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929\u20131958.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_77_2","article-title":"Inception-v4, Inception-ResNet and the impact of residual connections on learning","author":"Szegedy Christian","year":"2017","unstructured":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_78_2","article-title":"Learning spatiotemporal features with 3D convolutional networks","author":"Tran Du","year":"2015","unstructured":"Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3D convolutional networks. In ICCV.","journal-title":"ICCV"},{"key":"e_1_3_3_79_2","article-title":"Attention is all you need","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.","journal-title":"NIPS"},{"key":"e_1_3_3_80_2","article-title":"CIDEr: Consensus-based image description evaluation","author":"Vedantam Ramakrishna","year":"2015","unstructured":"Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_81_2","article-title":"Show and tell: A neural image caption generator","author":"Vinyals Oriol","year":"2015","unstructured":"Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In CVPR.","journal-title":"CVPR"},{"issue":"4","key":"e_1_3_3_82_2","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1109\/TPAMI.2016.2587640","article-title":"Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge","volume":"39","author":"Vinyals Oriol","year":"2017","unstructured":"Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2017. Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 4 (2017), 652\u2013663.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_3_83_2","article-title":"Controllable video captioning with POS sequence guidance based on gated fusion network","author":"Wang Bairui","year":"2019","unstructured":"Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, and Wei Liu. 2019. Controllable video captioning with POS sequence guidance based on gated fusion network. In ICCV.","journal-title":"ICCV"},{"key":"e_1_3_3_84_2","article-title":"Learning deep transformer models for machine translation","author":"Wang Qiang","year":"2019","unstructured":"Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In ACL.","journal-title":"ACL"},{"key":"e_1_3_3_85_2","article-title":"Multi-layer representation fusion for neural machine translation","author":"Wang Qiang","year":"2018","unstructured":"Qiang Wang, Fuxue Li, Tong Xiao, Yanyang Li, Yinqiao Li, and Jingbo Zhu. 2018. Multi-layer representation fusion for neural machine translation. In COLING.","journal-title":"COLING"},{"key":"e_1_3_3_86_2","article-title":"Pay less attention with lightweight and dynamic convolutions","author":"Wu Felix","year":"2019","unstructured":"Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In ICLR.","journal-title":"ICLR"},{"key":"e_1_3_3_87_2","article-title":"MSR-VTT: A large video description dataset for bridging video and language","author":"Xu Jun","year":"2016","unstructured":"Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. MSR-VTT: A large video description dataset for bridging video and language. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_88_2","article-title":"Show, attend and tell: Neural image caption generation with visual attention","author":"Xu Kelvin","year":"2015","unstructured":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML.","journal-title":"ICML"},{"issue":"1","key":"e_1_3_3_89_2","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1109\/TMM.2019.2924576","article-title":"STAT: Spatial-temporal attention mechanism for video captioning","volume":"22","author":"Yan Chenggang","year":"2020","unstructured":"Chenggang Yan, Yunbin Tu, Xingzheng Wang, Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. 2020. STAT: Spatial-temporal attention mechanism for video captioning. IEEE Transactions on Multimedia 22, 1 (2020), 229\u2013241.","journal-title":"IEEE Transactions on Multimedia"},{"key":"e_1_3_3_90_2","article-title":"Auto-encoding scene graphs for image captioning","author":"Yang Xu","year":"2019","unstructured":"Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019. Auto-encoding scene graphs for image captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_91_2","article-title":"Exploring visual relationship for image captioning","author":"Yao Ting","year":"2018","unstructured":"Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2018. Exploring visual relationship for image captioning. In ECCV.","journal-title":"ECCV"},{"key":"e_1_3_3_92_2","article-title":"Fixup initialization: Residual learning without normalization","author":"Zhang Hongyi","year":"2019","unstructured":"Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. 2019. Fixup initialization: Residual learning without normalization. In ICLR.","journal-title":"ICLR"},{"key":"e_1_3_3_93_2","article-title":"RSTNet: Captioning with adaptive attention on visual and non-visual words","author":"Zhang Xuying","year":"2021","unstructured":"Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. 2021. RSTNet: Captioning with adaptive attention on visual and non-visual words. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_94_2","article-title":"When radiology report generation meets knowledge graph","author":"Zhang Yixiao","year":"2020","unstructured":"Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan L. Yuille, and Daguang Xu. 2020. When radiology report generation meets knowledge graph. In AAAI.","journal-title":"AAAI"},{"key":"e_1_3_3_95_2","article-title":"Object relational graph with teacher-recommended learning for video captioning","author":"Zhang Ziqi","year":"2020","unstructured":"Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. 2020. Object relational graph with teacher-recommended learning for video captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_96_2","article-title":"Syntax-aware action targeting for video captioning","author":"Zheng Qi","year":"2020","unstructured":"Qi Zheng, Chaoyue Wang, and Dacheng Tao. 2020. Syntax-aware action targeting for video captioning. In CVPR.","journal-title":"CVPR"},{"key":"e_1_3_3_97_2","unstructured":"Tong Zheng Bei Li Huiwen Bao Tong Xiao and Jingbo Zhu. 2022. EIT: Enhanced interactive transformer. arXiv:2212.10197. Retrieved from https:\/\/arxiv.org\/abs\/2212.10197"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729536","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3729536","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:56:57Z","timestamp":1750298217000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729536"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,6]]},"references-count":96,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3729536"],"URL":"https:\/\/doi.org\/10.1145\/3729536","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,6]]},"assertion":[{"value":"2022-09-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}