{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,13]],"date-time":"2025-05-13T22:00:14Z","timestamp":1747173614982,"version":"3.40.5"},"reference-count":62,"publisher":"Cambridge University Press (CUP)","issue":"1","license":[{"start":{"date-parts":[[2021,11,24]],"date-time":"2021-11-24T00:00:00Z","timestamp":1637712000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Convolutional sequence to sequence (CNN seq2seq) models have met success in abstractive summarization. However, their outputs often contain repetitive word sequences and logical inconsistencies, limiting the practicality of their application. In this paper, we find the reasons behind the repetition problem in CNN-based abstractive summarization through observing the attention map between the summaries with repetition and their corresponding source documents and mitigate the repetition problem. We propose to reduce the repetition in summaries by attention filter mechanism (ATTF) and sentence-level backtracking decoder (SBD), which dynamically redistributes attention over the input sequence as the output sentences are generated. The ATTF can record previously attended locations in the source document directly and prevent the decoder from attending to these locations. The SBD prevents the decoder from generating similar sentences more than once via backtracking at test. The proposed model outperforms the baselines in terms of ROUGE score, repeatedness, and readability. The results show that this approach generates high-quality summaries with minimal repetition and makes the reading experience better.<\/jats:p>","DOI":"10.1017\/s1351324921000309","type":"journal-article","created":{"date-parts":[[2021,11,24]],"date-time":"2021-11-24T08:00:12Z","timestamp":1637740812000},"page":"81-109","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":1,"title":["Reducing repetition in convolutional abstractive summarization"],"prefix":"10.1017","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7241-120X","authenticated-orcid":false,"given":"Yizhu","family":"Liu","sequence":"first","affiliation":[]},{"given":"Xinyue","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Xusheng","family":"Luo","sequence":"additional","affiliation":[]},{"given":"Kenny Q.","family":"Zhu","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2021,11,24]]},"reference":[{"key":"S1351324921000309_ref44","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671927"},{"key":"S1351324921000309_ref45","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1044"},{"key":"S1351324921000309_ref54","unstructured":"Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, L. and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998\u20136008."},{"key":"S1351324921000309_ref15","unstructured":"Dong, L. , Yang, N. , Wang, W. , Wei, F. , Liu, X. , Wang, Y. , Gao, J. , Zhou, M. and Hon, H. (2019). Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 13042\u201313054."},{"key":"S1351324921000309_ref35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1444"},{"key":"S1351324921000309_ref34","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1500"},{"key":"S1351324921000309_ref6","unstructured":"Briscoe, T. (1996). The syntax and semantics of punctuation and its use in interpretation. In Proceedings of the Association for Computational Linguistics Workshop on Punctuation, pp. 1\u20137."},{"key":"S1351324921000309_ref33","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449906"},{"key":"S1351324921000309_ref31","unstructured":"Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74\u201381, Barcelona, Spain."},{"key":"S1351324921000309_ref27","unstructured":"Li, W. , He, L. and Zhuge, H. (2016). Abstractive news summarization based on event semantic link network. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan. Association for Computer Linguistics, pp. 236\u2013246."},{"key":"S1351324921000309_ref11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1063"},{"key":"S1351324921000309_ref18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1443"},{"key":"S1351324921000309_ref12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1012"},{"key":"S1351324921000309_ref30","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00273"},{"key":"S1351324921000309_ref48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1323"},{"key":"S1351324921000309_ref40","doi-asserted-by":"crossref","unstructured":"Pallotta, V. , Delmonte, R. and Bristot, A. (2009). Abstractive summarization of voice communications. In Human Language Technology. Challenges for Computer Science and Linguistics - 4th Language and Technology Conference, LTC 2009, Poznan, Poland, November 6-8, 2009, Revised Selected Papers. Springer, pp. 291\u2013302.","DOI":"10.1007\/978-3-642-20095-3_27"},{"key":"S1351324921000309_ref47","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1099"},{"key":"S1351324921000309_ref13","unstructured":"Dauphin, Y.N. , Fan, A. , Auli, M. and Grangier, D. (2017). Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. PMLR, pp. 933\u2013941."},{"key":"S1351324921000309_ref61","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.552"},{"key":"S1351324921000309_ref56","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1207"},{"key":"S1351324921000309_ref38","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324918000414"},{"key":"S1351324921000309_ref3","unstructured":"Bai, S. , Kolter, J.Z. and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv, abs\/1803.01271."},{"key":"S1351324921000309_ref62","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1100"},{"key":"S1351324921000309_ref46","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2014.02.001"},{"key":"S1351324921000309_ref60","doi-asserted-by":"publisher","DOI":"10.3390\/app9081665"},{"key":"S1351324921000309_ref42","unstructured":"Paulus, R. , Xiong, C. and Socher, R. (2018). A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net."},{"key":"S1351324921000309_ref8","doi-asserted-by":"crossref","unstructured":"Carenini, G. and Cheung, J.C.K. (2008). Extractive vs. NLG-based abstractive summarization of evaluative text: The effect of corpus controversiality. In INLG 2008 - Proceedings of the Fifth International Natural Language Generation Conference, June 12\u201314, 2008, Salt Fork, Ohio, USA. The Association for Computer Linguistics.","DOI":"10.3115\/1708322.1708330"},{"key":"S1351324921000309_ref24","unstructured":"Kulesza, A. and Taskar, B. (2011). k-DPPs: Fixed-size determinantal point processes. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011. Omnipress, pp. 1193\u20131200."},{"key":"S1351324921000309_ref37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K16-1028"},{"key":"S1351324921000309_ref5","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324914000199"},{"key":"S1351324921000309_ref1","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2017.081052"},{"key":"S1351324921000309_ref9","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1150"},{"key":"S1351324921000309_ref20","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"S1351324921000309_ref14","unstructured":"Devlin, J. , Chang, M. , Lee, K. and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers). Association for Computational Linguistics, pp. 4171\u20134186."},{"key":"S1351324921000309_ref41","unstructured":"Pascanu, R. , Mikolov, T. and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013. JMLR.org, pp. 1310\u20131318."},{"key":"S1351324921000309_ref19","doi-asserted-by":"crossref","unstructured":"Grusky, M. , Naaman, M. and Artzi, Y. (2018). Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers). Association for Computational Linguistics, pp. 708\u2013719.","DOI":"10.18653\/v1\/N18-1065"},{"key":"S1351324921000309_ref7","doi-asserted-by":"publisher","DOI":"10.1142\/S0218001493000339"},{"key":"S1351324921000309_ref53","unstructured":"Vaswani, A. , Bengio, S. , Brevdo, E. , Chollet, F. , Gomez, A. N. , Gouws, S. , Jones, L. , Kaiser, L. , Kalchbrenner, N. , Parmar, N. , Sepassi, R. , Shazeer, N. and Uszkoreit, J. (2018). Tensor2tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas, AMTA 2018, Boston, MA, USA, March 17-21, 2018 - Volume 1: Research Papers. Association for Machine Translation in the Americas, pp. 193\u2013199."},{"key":"S1351324921000309_ref17","unstructured":"Gehring, J. , Auli, M. , Grangier, D. , Yarats, D. and Dauphin, Y. N. (2017). Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. PMLR, pp. 1243\u20131252."},{"key":"S1351324921000309_ref39","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2018.12.006"},{"key":"S1351324921000309_ref50","unstructured":"Sutskever, I. , Martens, J. , Dahl, G. E. and Hinton, G. E. (2013). On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013. JMLR.org, pp. 1139\u20131147."},{"key":"S1351324921000309_ref49","doi-asserted-by":"publisher","DOI":"10.1145\/3419106"},{"key":"S1351324921000309_ref22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1440"},{"key":"S1351324921000309_ref2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-5402"},{"key":"S1351324921000309_ref29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1441"},{"key":"S1351324921000309_ref32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2027"},{"key":"S1351324921000309_ref36","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-1809"},{"key":"S1351324921000309_ref43","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.217"},{"key":"S1351324921000309_ref51","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-2047"},{"key":"S1351324921000309_ref55","article-title":"Extractive summarization: Limits, compression, generalized model and heuristics","volume":"21","author":"Verma","year":"2017","journal-title":"Computaci\u00f3n y Sistemas"},{"key":"S1351324921000309_ref58","unstructured":"Zhang, J. , Zhao, Y. , Saleh, M. and Liu, P. J. (2020). PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research. PMLR, pp. 11328\u201311339."},{"key":"S1351324921000309_ref59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1499"},{"key":"S1351324921000309_ref26","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K19-1077"},{"key":"S1351324921000309_ref52","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1108"},{"key":"S1351324921000309_ref25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"S1351324921000309_ref23","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2019.8682418"},{"key":"S1351324921000309_ref57","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-017-1042-4"},{"key":"S1351324921000309_ref10","unstructured":"Chen, Q. , Zhu, X. , Ling, Z. , Wei, S. and Jiang, H. (2016). Distraction-based neural networks for modeling document. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016. IJCAI\/AAAI Press, pp. 2754\u20132760."},{"key":"S1351324921000309_ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-2706"},{"key":"S1351324921000309_ref4","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1153"},{"key":"S1351324921000309_ref28","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1205"},{"key":"S1351324921000309_ref21","unstructured":"Hermann, K.M. , Kocisk\u00fd, T. , Grefenstette, E. , Espeholt, L. , Kay, W. , Suleyman, M. and Blunsom, P. (2015). Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1693\u20131701."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324921000309","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T09:05:38Z","timestamp":1675155938000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324921000309\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,24]]},"references-count":62,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1]]}},"alternative-id":["S1351324921000309"],"URL":"https:\/\/doi.org\/10.1017\/s1351324921000309","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"type":"print","value":"1351-3249"},{"type":"electronic","value":"1469-8110"}],"subject":[],"published":{"date-parts":[[2021,11,24]]},"assertion":[{"value":"\u00a9 The Author(s), 2021. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}}]}}