{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T15:23:51Z","timestamp":1792509831341,"version":"4.5.0"},"reference-count":57,"publisher":"MIT Press","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p>Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2, and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.<\/jats:p>","DOI":"10.1162\/tacl_a_00313","type":"journal-article","created":{"date-parts":[[2020,6,17]],"date-time":"2020-06-17T14:25:54Z","timestamp":1592403954000},"page":"264-280","source":"Crossref","is-referenced-by-count":203,"title":["Leveraging Pre-trained Checkpoints for Sequence Generation Tasks"],"prefix":"10.1162","volume":"8","author":[{"given":"Sascha","family":"Rothe","sequence":"first","affiliation":[{"name":"Google Research."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shashi","family":"Narayan","sequence":"additional","affiliation":[{"name":"Google Research."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aliaksei","family":"Severyn","sequence":"additional","affiliation":[{"name":"Google Research."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","first-page":"719","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Aharoni Roee","year":"2018"},{"key":"bib2","doi-asserted-by":"crossref","first-page":"732","DOI":"10.18653\/v1\/D18-1080","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Botha Jan A.","year":"2018"},{"key":"bib3","author":"Chan William","year":"2019","journal-title":"CoRR"},{"key":"bib4","first-page":"7057","volume-title":"Advances in Neural Information Processing Systems 32","author":"Conneau Alexis","year":"2019"},{"key":"bib5","first-page":"3079","volume-title":"Advances in Neural Information Processing Systems 28","author":"Dai Andrew M.","year":"2015"},{"key":"bib6","first-page":"2978","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Dai Zihang","year":"2019"},{"key":"bib7","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin Jacob","year":"2019"},{"key":"bib8","first-page":"13042","volume-title":"Advances in Neural Information Processing Systems 32","author":"Li Dong","year":"2019"},{"key":"bib9","doi-asserted-by":"crossref","first-page":"489","DOI":"10.18653\/v1\/D18-1045","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Edunov Sergey","year":"2018"},{"key":"bib10","first-page":"625","volume":"11","author":"Erhan Dumitru","year":"2010","journal-title":"Journal of Machine Learning Research"},{"key":"bib11","first-page":"1243","volume-title":"Proceedings of the 34th International Conference on Machine Learning","volume":"70","author":"Gehring Jonas","year":"2017"},{"key":"bib12","doi-asserted-by":"crossref","first-page":"4098","DOI":"10.18653\/v1\/D18-1443","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Gehrmann Sebastian","year":"2018"},{"key":"bib13","first-page":"3443","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Geva Mor","year":"2019"},{"key":"bib14","first-page":"1631","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","author":"Jiatao Gu","year":"2016"},{"key":"bib15","author":"Hansen Nikolaus","year":"2016","journal-title":"CoRR"},{"key":"bib16","first-page":"7944","volume-title":"Advances in Neural Information Processing Systems 31","author":"He Tianyu","year":"2018"},{"key":"bib17","author":"Hendrycks Dan","year":"2016","journal-title":"CoRR"},{"key":"bib18","first-page":"1693","volume-title":"Advances in Neural Information Processing Systems 28","author":"Hermann Karl Moritz","year":"2015"},{"key":"bib19","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"bib20","first-page":"328","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Howard Jeremy","year":"2018"},{"key":"bib21","first-page":"873","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics","author":"Huang Eric","year":"2012"},{"key":"bib22","doi-asserted-by":"crossref","first-page":"3651","DOI":"10.18653\/v1\/P19-1356","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Jawahar Ganesh","year":"2019"},{"key":"bib23","author":"Khandelwal Urvashi","year":"2019","journal-title":"CoRR"},{"key":"bib24","first-page":"2519","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Kim Byeongchang","year":"2019"},{"key":"bib25","doi-asserted-by":"crossref","first-page":"66","DOI":"10.18653\/v1\/D18-2012","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Kudo Taku","year":"2018"},{"key":"bib26","author":"Lewis Mike","year":"2019","journal-title":"CoRR"},{"key":"bib27","first-page":"150","volume-title":"Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics","author":"Lin Chin Yew","year":"2003"},{"key":"bib28","author":"Liu Yang","year":"2019","journal-title":"CoRR"},{"key":"bib29","author":"Liu Yinhan","year":"2019","journal-title":"CoRR"},{"key":"bib30","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781107337855","volume-title":"Best-worst scaling: Theory, methods and applications","author":"Louviere Jordan J.","year":"2015"},{"key":"bib31","unstructured":"Jordan J. Louviere and George G. Woodworth. 1991. Best-worst scaling: A model for the largest difference judgments. University of Alberta, Working Paper."},{"key":"bib32","author":"Mikolov Tomas","year":"2013","journal-title":"CoRR"},{"key":"bib33","first-page":"95","volume-title":"Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction","author":"Napoles Courtney","year":"2012"},{"key":"bib34","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.18653\/v1\/D18-1206","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Narayan Shashi","year":"2018"},{"key":"bib35","first-page":"1747","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Narayan Shashi","year":"2018"},{"key":"bib36","first-page":"606","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Narayan Shashi","year":"2017"},{"key":"bib37","first-page":"1532","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","author":"Pennington Jeffrey","year":"2014"},{"key":"bib38","first-page":"2227","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Peters Matthew","year":"2018"},{"key":"bib39","first-page":"7","volume-title":"Proceedings of the 4th Workshop on Representation Learning for NLP","author":"Peters Matthew E.","year":"2019"},{"key":"bib40","unstructured":"Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI."},{"key":"bib41","unstructured":"Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI."},{"key":"bib42","first-page":"784","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Rajpurkar Pranav","year":"2018"},{"key":"bib43","first-page":"383","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Ramachandran Prajit","year":"2017"},{"key":"bib44","first-page":"1793","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","author":"Rothe Sascha","year":"2015"},{"key":"bib45","first-page":"1073","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","author":"See Abigail","year":"2017"},{"key":"bib46","first-page":"1715","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","author":"Sennrich Rico","year":"2016"},{"key":"bib47","first-page":"464","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Shaw Peter","year":"2018"},{"key":"bib48","first-page":"5926","volume-title":"Proceedings of the 36th International Conference on Machine Learning","volume":"97","author":"Song Kaitao","year":"2019"},{"key":"bib49","doi-asserted-by":"crossref","first-page":"4593","DOI":"10.18653\/v1\/P19-1452","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Tenney Ian","year":"2019"},{"key":"bib50","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems 30","author":"Vaswani Ashish","year":"2017"},{"key":"bib51","doi-asserted-by":"crossref","first-page":"4465","DOI":"10.18653\/v1\/P19-1439","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Wang Alex","year":"2019"},{"key":"bib52","author":"Wang Ran","year":"2019","journal-title":"CoRR"},{"key":"bib53","first-page":"1112","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Williams Adina","year":"2018"},{"key":"bib54","author":"Yonghui Wu","year":"2016","journal-title":"CoRR"},{"key":"bib55","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1162\/tacl_a_00107","volume":"4","author":"Wei Xu","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"bib56","author":"Yang Zhilin","year":"2019","journal-title":"CoRR"},{"key":"bib57","doi-asserted-by":"crossref","first-page":"93","DOI":"10.18653\/v1\/D18-1009","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Zellers Rowan","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00313","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T16:39:36Z","timestamp":1615567176000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96450"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":57,"alternative-id":["10.1162\/tacl_a_00313"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00313","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}