{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T13:27:04Z","timestamp":1784640424615,"version":"3.55.0"},"reference-count":41,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,9,21]],"date-time":"2022-09-21T00:00:00Z","timestamp":1663718400000},"content-version":"vor","delay-in-days":263,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,9,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>When generating text from probabilistic models, the chosen decoding strategy has a profound effect on the resulting text. Yet the properties elicited by various decoding strategies do not always transfer across natural language generation tasks. For example, while mode-seeking methods like beam search perform remarkably well for machine translation, they have been observed to lead to incoherent and repetitive text in story generation. Despite such observations, the effectiveness of decoding strategies is often assessed on only a single task. This work\u2014in contrast\u2014provides a comprehensive analysis of the interaction between language generation tasks and decoding strategies. Specifically, we measure changes in attributes of generated text as a function of both decoding strategy and task using human and automatic evaluation. Our results reveal both previously observed and novel findings. For example, the nature of the diversity\u2013quality trade-off in language generation is very task-specific; the length bias often attributed to beam search is not constant across tasks. https:\/\/github.com\/gianwiher\/decoding-NLG<\/jats:p>","DOI":"10.1162\/tacl_a_00502","type":"journal-article","created":{"date-parts":[[2022,9,21]],"date-time":"2022-09-21T18:03:56Z","timestamp":1663783436000},"page":"997-1012","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":28,"title":["On Decoding Strategies for Neural Text Generators"],"prefix":"10.1162","volume":"10","author":[{"given":"Gian","family":"Wiher","sequence":"first","affiliation":[{"name":"ETH Z\u00fcrich, Switzerland. gian.wiher@inf.ethz.ch"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clara","family":"Meister","sequence":"additional","affiliation":[{"name":"ETH Z\u00fcrich, Switzerland. clara.meister@inf.ethz.ch"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryan","family":"Cotterell","sequence":"additional","affiliation":[{"name":"ETH Z\u00fcrich, Switzerland. ryan.cotterell@inf.ethz.ch"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,9,19]]},"reference":[{"key":"2022092118021327800_bib1","volume-title":"Mathematical Statistics: Basic Ideas and Selected Topics","author":"Bickel","year":"1977","edition":"2"},{"key":"2022092118021327800_bib2","article-title":"Empirical analysis of beam search performance degradation in neural sequence models","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Cohen","year":"2019"},{"key":"2022092118021327800_bib3","doi-asserted-by":"publisher","first-page":"166","DOI":"10.18653\/v1\/2021.gem-1.16","article-title":"Decoding methods for neural narrative generation","volume-title":"Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)","author":"DeLucia","year":"2021"},{"key":"2022092118021327800_bib4","doi-asserted-by":"publisher","first-page":"376","DOI":"10.3115\/v1\/W14-3348","article-title":"Meteor universal: Language specific translation evaluation for any target language","volume-title":"Proceedings of the Ninth Workshop on Statistical Machine Translation","author":"Denkowski","year":"2014"},{"key":"2022092118021327800_bib5","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2022092118021327800_bib6","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1007\/978-3-030-29135-8_7","article-title":"The second conversational intelligence challenge (ConvAI2)","volume-title":"The NeurIPS \u201918 Competition","author":"Dinan","year":"2020"},{"key":"2022092118021327800_bib7","doi-asserted-by":"publisher","first-page":"4506","DOI":"10.18653\/v1\/2020.coling-main.398","article-title":"Is MAP decoding all you need? The inadequacy of the mode in neural machine translation","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8\u201313, 2020","author":"Eikema","year":"2020"},{"key":"2022092118021327800_bib8","doi-asserted-by":"crossref","first-page":"889","DOI":"10.18653\/v1\/P18-1082","article-title":"Hierarchical neural story generation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Fan","year":"2018"},{"key":"2022092118021327800_bib9","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00437","article-title":"Experts, errors, and context: A large-scale study of human evaluation for machine translation","volume":"abs\/2104.14478","author":"Freitag","year":"2021","journal-title":"CoRR"},{"key":"2022092118021327800_bib10","article-title":"The curious case of neural text degeneration","author":"Holtzman","year":"2020","journal-title":"International Conference on Learning Representations"},{"key":"2022092118021327800_bib11","doi-asserted-by":"publisher","first-page":"3752","DOI":"10.18653\/v1\/P19-1365","article-title":"Comparison of diverse decoding methods from conditional language models","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Ippolito","year":"2019"},{"key":"2022092118021327800_bib12","doi-asserted-by":"publisher","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lewis","year":"2020"},{"key":"2022092118021327800_bib13","first-page":"110","article-title":"A diversity-promoting objective function for neural conversation models","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Li","year":"2016"},{"key":"2022092118021327800_bib14","article-title":"Mutual information and diverse decoding improve neural machine translation","volume":"abs\/1601.00372","author":"Li","year":"2016","journal-title":"CoRR"},{"key":"2022092118021327800_bib15","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2022092118021327800_bib16","doi-asserted-by":"publisher","first-page":"212","DOI":"10.18653\/v1\/W18-6322","article-title":"Correcting length bias in neural machine translation","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Murray","year":"2018"},{"key":"2022092118021327800_bib17","first-page":"334","article-title":"A systematic characterization of sampling algorithms for open-ended language generation","volume-title":"Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing","author":"Nadeem","year":"2020"},{"key":"2022092118021327800_bib18","doi-asserted-by":"crossref","first-page":"314","DOI":"10.18653\/v1\/W19-5333","article-title":"Facebook FAIR\u2019s WMT19 news translation task submission","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Ng","year":"2019"},{"key":"2022092118021327800_bib19","first-page":"3956","article-title":"Analyzing uncertainty in neural machine translation","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Ott","year":"2018"},{"key":"2022092118021327800_bib20","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2022092118021327800_bib21","doi-asserted-by":"publisher","first-page":"186","DOI":"10.18653\/v1\/W18-6319","article-title":"A call for clarity in reporting BLEU scores","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Post","year":"2018"},{"key":"2022092118021327800_bib22","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019"},{"key":"2022092118021327800_bib23","doi-asserted-by":"crossref","first-page":"2685","DOI":"10.18653\/v1\/2020.emnlp-main.213","article-title":"COMET: A neural framework for MT evaluation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Rei","year":"2020"},{"key":"2022092118021327800_bib24","doi-asserted-by":"crossref","first-page":"379","DOI":"10.18653\/v1\/D15-1044","article-title":"A neural attention model for abstractive sentence summarization","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Rush","year":"2015"},{"key":"2022092118021327800_bib25","doi-asserted-by":"publisher","first-page":"7881","DOI":"10.18653\/v1\/2020.acl-main.704","article-title":"BLEURT: Learning robust metrics for text generation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Sellam","year":"2020"},{"key":"2022092118021327800_bib26","doi-asserted-by":"publisher","first-page":"3288","DOI":"10.1609\/aaai.v31i1.10984","article-title":"Multiresolution recurrent neural networks: An application to dialogue response generation","volume-title":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence","author":"Serban","year":"2017"},{"key":"2022092118021327800_bib27","doi-asserted-by":"publisher","first-page":"2210","DOI":"10.18653\/v1\/D17-1235","article-title":"Generating high-quality and informative conversation responses with sequence-to-sequence models","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Shao","year":"2017"},{"key":"2022092118021327800_bib28","doi-asserted-by":"publisher","first-page":"3356","DOI":"10.18653\/v1\/D19-1331","article-title":"On NMT search errors and model errors: Cat got your tongue?","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Stahlberg","year":"2019"},{"key":"2022092118021327800_bib29","doi-asserted-by":"publisher","first-page":"202","DOI":"10.3115\/v1\/D14-1025","article-title":"Fitting sentence level translation evaluation with many dense features","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Stanojevi\u0107","year":"2014"},{"key":"2022092118021327800_bib30","article-title":"Sequence to sequence learning with neural networks","volume-title":"Advances in Neural Information Processing Systems","author":"Sutskever","year":"2014"},{"key":"2022092118021327800_bib31","doi-asserted-by":"publisher","first-page":"2818","DOI":"10.1109\/CVPR.2016.308","article-title":"Rethinking the inception architecture for computer vision","volume-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Szegedy","year":"2016"},{"key":"2022092118021327800_bib32","doi-asserted-by":"publisher","first-page":"101151","DOI":"10.1016\/j.csl.2020.101151","article-title":"Human evaluation of automatically generated text: Current trends and best practice guidelines","volume":"67","author":"van der Lee","year":"2021","journal-title":"Computer Speech & Language"},{"key":"2022092118021327800_bib33","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2022092118021327800_bib34","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12340","article-title":"Diverse beam search for improved description of complex scenes","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","author":"Vijayakumar","year":"2018"},{"key":"2022092118021327800_bib35","doi-asserted-by":"publisher","first-page":"5553","DOI":"10.18653\/v1\/2020.emnlp-main.448","article-title":"Consistency of a recurrent language model with respect to incomplete decoding","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Welleck","year":"2020"},{"key":"2022092118021327800_bib36","article-title":"Neural text generation with unlikelihood training","volume-title":"8th International Conference on Learning Representations, ICLR","author":"Welleck","year":"2020"},{"key":"2022092118021327800_bib37","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2022092118021327800_bib38","first-page":"25","article-title":"Trading off diversity and quality in natural language generation","volume-title":"Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)","author":"Zhang","year":"2021"},{"key":"2022092118021327800_bib39","article-title":"Generating informative and diverse conversational responses via adversarial information maximization","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang","year":"2018"},{"key":"2022092118021327800_bib40","doi-asserted-by":"publisher","first-page":"270","DOI":"10.18653\/v1\/2020.acl-demos.30","article-title":"DIALOGPT: Large-scale generative pre-training for conversational response generation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Zhang","year":"2020"},{"key":"2022092118021327800_bib41","first-page":"1097","article-title":"Texygen: A benchmarking platform for text generation models","volume-title":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval","author":"Zhu","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00502\/2043936\/tacl_a_00502.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00502\/2043936\/tacl_a_00502.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,21]],"date-time":"2022-09-21T18:04:42Z","timestamp":1663783482000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00502\/113024\/On-Decoding-Strategies-for-Neural-Text-Generators"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":41,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00502","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}