{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T17:05:13Z","timestamp":1792688713757,"version":"4.5.0"},"reference-count":64,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,3,23]],"date-time":"2022-03-23T00:00:00Z","timestamp":1647993600000},"content-version":"vor","delay-in-days":81,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,3,25]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.1<\/jats:p>","DOI":"10.1162\/tacl_a_00461","type":"journal-article","created":{"date-parts":[[2022,3,23]],"date-time":"2022-03-23T15:20:07Z","timestamp":1648048807000},"page":"291-306","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":204,"title":["ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models"],"prefix":"10.1162","volume":"10","author":[{"given":"Linting","family":"Xue","sequence":"first","affiliation":[{"name":"Google Research, USA. lintingx@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditya","family":"Barua","sequence":"additional","affiliation":[{"name":"Google Research, USA. adityabarua@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Noah","family":"Constant","sequence":"additional","affiliation":[{"name":"Google Research, USA. nconstant@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rami","family":"Al-Rfou","sequence":"additional","affiliation":[{"name":"Google Research, USA. rmyeid@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sharan","family":"Narang","sequence":"additional","affiliation":[{"name":"Google Research, USA. sharannarang@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mihir","family":"Kale","sequence":"additional","affiliation":[{"name":"Google Research, USA. mihirkale@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adam","family":"Roberts","sequence":"additional","affiliation":[{"name":"Google Research, USA. adarob@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Colin","family":"Raffel","sequence":"additional","affiliation":[{"name":"Google Research, USA. craffel@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,3,25]]},"reference":[{"key":"2022033118513588900_bib1","first-page":"1638","article-title":"Contextual string embeddings for sequence labeling","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Akbik","year":"2018"},{"issue":"01","key":"2022033118513588900_bib2","doi-asserted-by":"publisher","first-page":"3159","DOI":"10.1609\/aaai.v33i01.33013159","article-title":"Character-level language modeling with deeper self-attention","volume":"33","author":"Al-Rfou","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2022033118513588900_bib3","doi-asserted-by":"crossref","first-page":"4623","DOI":"10.18653\/v1\/2020.acl-main.421","article-title":"On the cross-lingual transferability of monolingual representations","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Artetxe","year":"2020"},{"key":"2022033118513588900_bib4","doi-asserted-by":"crossref","first-page":"6588","DOI":"10.18653\/v1\/2020.coling-main.579","article-title":"Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Caswell","year":"2020"},{"key":"2022033118513588900_bib5","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.emnlp-main.549","article-title":"Question directed graph attention network for numerical reasoning over text","author":"Chen","year":"2020"},{"key":"2022033118513588900_bib6","doi-asserted-by":"crossref","first-page":"4295","DOI":"10.18653\/v1\/D18-1461","article-title":"Revisiting character-based neural machine translation with capacity and compression","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Cherry","year":"2018"},{"key":"2022033118513588900_bib7","article-title":"Bridging the gap for tokenizer-free language models","author":"Choe","year":"2019","journal-title":"CoRR"},{"key":"2022033118513588900_bib8","article-title":"Hierarchical multiscale recurrent neural networks","volume-title":"5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\u201326, 2017, Conference Track Proceedings","author":"Chung","year":"2017"},{"key":"2022033118513588900_bib9","doi-asserted-by":"crossref","first-page":"1693","DOI":"10.18653\/v1\/P16-1160","article-title":"A character-level decoder without explicit segmentation for neural machine translation","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Chung","year":"2016"},{"key":"2022033118513588900_bib10","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1162\/tacl_a_00317","article-title":"TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages","volume":"8","author":"Clark","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022033118513588900_bib11","article-title":"CANINE: pre-training an efficient tokenization-free encoder for language representation","author":"Clark","year":"2021","journal-title":"CoRR"},{"key":"2022033118513588900_bib12","doi-asserted-by":"crossref","first-page":"2475","DOI":"10.18653\/v1\/D18-1269","article-title":"XNLI: Evaluating cross-lingual sentence representations","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Conneau","year":"2018"},{"key":"2022033118513588900_bib13","doi-asserted-by":"crossref","first-page":"154","DOI":"10.18653\/v1\/W17-4123","article-title":"Byte-based neural machine translation","volume-title":"Proceedings of the First Workshop on Subword and Character Level Models in NLP","author":"Costa-juss\u00e0","year":"2017"},{"key":"2022033118513588900_bib14","first-page":"2368","article-title":"DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Dua","year":"2019"},{"key":"2022033118513588900_bib15","first-page":"1504","article-title":"One size does not fit all: Comparing NMT representations of different granularities","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Durrani","year":"2019"},{"key":"2022033118513588900_bib16","doi-asserted-by":"crossref","first-page":"6903","DOI":"10.18653\/v1\/2020.coling-main.609","article-title":"CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Boukkouri","year":"2020"},{"key":"2022033118513588900_bib17","article-title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","author":"Fedus","year":"2021","journal-title":"CoRR"},{"key":"2022033118513588900_bib18","first-page":"1184","article-title":"Towards continual learning for multilingual machine translation via vocabulary substitution","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Garcia","year":"2021"},{"key":"2022033118513588900_bib19","article-title":"The GEM benchmark: Natural language generation, its evaluation and metrics","author":"Gehrmann","year":"2021","journal-title":"CoRR"},{"key":"2022033118513588900_bib20","first-page":"1296","article-title":"Multilingual language processing from bytes","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Gillick","year":"2016"},{"key":"2022033118513588900_bib21","doi-asserted-by":"crossref","first-page":"40","DOI":"10.18653\/v1\/2020.sigmorphon-1.2","article-title":"The SIGMORPHON 2020 shared task on multilingual grapheme-to-phoneme conversion","volume-title":"Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology","author":"Gorman","year":"2020"},{"key":"2022033118513588900_bib22","article-title":"Generating sequences with recurrent neural networks","author":"Graves","year":"2013","journal-title":"CoRR"},{"key":"2022033118513588900_bib23","article-title":"Adaptive computation time for recurrent neural networks","author":"Graves","year":"2016","journal-title":"CoRR"},{"key":"2022033118513588900_bib24","article-title":"Hypernetworks","volume-title":"5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24\u201326, 2017, Conference Track Proceedings","author":"Ha","year":"2017"},{"key":"2022033118513588900_bib25","doi-asserted-by":"crossref","first-page":"258","DOI":"10.18653\/v1\/W18-5428","article-title":"Interpreting word-level hidden state behaviour of character-level LSTM language models","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Hiebert","year":"2018"},{"key":"2022033118513588900_bib26","first-page":"4411","article-title":"XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Junjie","year":"2020"},{"key":"2022033118513588900_bib27","article-title":"Exploring the limits of language modeling","author":"J\u00f3zefowicz","year":"2016","journal-title":"CoRR"},{"key":"2022033118513588900_bib28","article-title":"Neural machine translation in linear time","author":"Kalchbrenner","year":"2016","journal-title":"CoRR"},{"key":"2022033118513588900_bib29","article-title":"Scaling laws for neural language models","author":"Kaplan","year":"2020","journal-title":"CoRR"},{"key":"2022033118513588900_bib30","first-page":"2741","article-title":"Character-aware neural language models","volume-title":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","author":"Kim","year":"2016"},{"key":"2022033118513588900_bib31","doi-asserted-by":"crossref","first-page":"66","DOI":"10.18653\/v1\/P18-1007","article-title":"Subword regularization: Improving neural network translation models with multiple subword candidates","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Kudo","year":"2018"},{"key":"2022033118513588900_bib32","doi-asserted-by":"crossref","first-page":"66","DOI":"10.18653\/v1\/D18-2012","article-title":"SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Kudo","year":"2018"},{"key":"2022033118513588900_bib33","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1162\/tacl_a_00067","article-title":"Fully character-level neural machine translation without explicit segmentation","volume":"5","author":"Lee","year":"2017","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022033118513588900_bib34","doi-asserted-by":"crossref","first-page":"7315","DOI":"10.18653\/v1\/2020.acl-main.653","article-title":"MLQA: Evaluating cross-lingual extractive question answering","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lewis","year":"2020"},{"key":"2022033118513588900_bib35","first-page":"5621","article-title":"Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes","volume-title":"ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Bo","year":"2019"},{"key":"2022033118513588900_bib36","article-title":"Character-based neural machine translation","author":"Ling","year":"2015","journal-title":"CoRR"},{"key":"2022033118513588900_bib37","article-title":"On the state of the art of evaluation in neural language models","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 \u2013 May 3, 2018, Conference Track Proceedings","author":"Melis","year":"2018"},{"key":"2022033118513588900_bib38","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.18653\/v1\/D18-1206","article-title":"Don\u2019t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Narayan","year":"2018"},{"key":"2022033118513588900_bib39","doi-asserted-by":"crossref","first-page":"1946","DOI":"10.18653\/v1\/P17-1178","article-title":"Cross-lingual name tagging and linking for 282 languages","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Pan","year":"2017"},{"key":"2022033118513588900_bib40","doi-asserted-by":"crossref","first-page":"63","DOI":"10.18653\/v1\/2020.sigmorphon-1.4","article-title":"One-size-fits-all multilingual models","volume-title":"Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology","author":"Peters","year":"2020"},{"key":"2022033118513588900_bib41","first-page":"2227","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Peters","year":"2018"},{"key":"2022033118513588900_bib42","doi-asserted-by":"crossref","first-page":"5582","DOI":"10.18653\/v1\/P19-1561","article-title":"Combating adversarial misspellings with robust word recognition","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Pruthi","year":"2019"},{"issue":"140","key":"2022033118513588900_bib43","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2022033118513588900_bib44","first-page":"2413","article-title":"Processing South Asian languages written in the Latin script: The dakshina dataset","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Roark","year":"2020"},{"key":"2022033118513588900_bib45","doi-asserted-by":"crossref","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with subword units","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2022033118513588900_bib46","first-page":"181","article-title":"Neural machine translation without embeddings","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Shaham","year":"2021"},{"key":"2022033118513588900_bib47","article-title":"Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT","author":"Sun","year":"2020","journal-title":"CoRR"},{"key":"2022033118513588900_bib48","first-page":"1017","article-title":"Generating text with recurrent neural networks","volume-title":"Proceedings of the 28th International Conference on International Conference on Machine Learning","author":"Sutskever","year":"2011"},{"key":"2022033118513588900_bib49","article-title":"Efficient transformers: A survey","author":"Yi","year":"2020","journal-title":"CoRR"},{"key":"2022033118513588900_bib50","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2022033118513588900_bib51","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18653\/v1\/2020.sigmorphon-1.1","article-title":"SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflection","volume-title":"Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology","author":"Vylomova","year":"2020"},{"key":"2022033118513588900_bib52","article-title":"SuperGLUE: A stickier benchmark for general-purpose language understanding systems","volume-title":"Advances in Neural Information Processing Systems","author":"Wang","year":"2019"},{"key":"2022033118513588900_bib53","article-title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","volume-title":"International Conference on Learning Representations","author":"Wang","year":"2019"},{"issue":"05","key":"2022033118513588900_bib54","doi-asserted-by":"publisher","first-page":"9154","DOI":"10.1609\/aaai.v34i05.6451","article-title":"Neural machine translation with byte-level subwords","volume":"34","author":"Wang","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2022033118513588900_bib55","article-title":"Training multilingual pre-trained language model with byte-level subwords","author":"Wei","year":"2021","journal-title":"CoRR"},{"key":"2022033118513588900_bib56","first-page":"11170","article-title":"Which transformer architecture fits my data? a vocabulary bottleneck in self-attention","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Wies","year":"2021"},{"key":"2022033118513588900_bib57","article-title":"Google\u2019s neural machine translation system: Bridging the gap between human and machine translation","author":"Yonghui","year":"2016","journal-title":"CoRR"},{"key":"2022033118513588900_bib58","doi-asserted-by":"crossref","first-page":"5020","DOI":"10.18653\/v1\/P19-1496","article-title":"TWEETQA: A social media focused question answering dataset","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Xiong","year":"2019"},{"key":"2022033118513588900_bib59","first-page":"483","article-title":"mT5: A massively multilingual pre-trained text-to-text transformer","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Xue","year":"2021"},{"key":"2022033118513588900_bib60","doi-asserted-by":"crossref","first-page":"3687","DOI":"10.18653\/v1\/D19-1382","article-title":"PAWS-X: A cross-lingual adversarial dataset for paraphrase identification","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Yang","year":"2019"},{"key":"2022033118513588900_bib61","first-page":"70","article-title":"Ensemble self-training for low-resource languages: Grapheme-to-phoneme conversion and morphological inflection","volume-title":"Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology","author":"Xiang","year":"2020"},{"key":"2022033118513588900_bib62","first-page":"11328","article-title":"PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Zhang","year":"2020"},{"key":"2022033118513588900_bib63","article-title":"Character-level convolutional networks for text classification","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang","year":"2015"},{"key":"2022033118513588900_bib64","first-page":"4189","article-title":"Recurrent highway networks","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Zilly","year":"2017"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00461\/2004058\/tacl_a_00461.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00461\/2004058\/tacl_a_00461.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T19:53:50Z","timestamp":1648756430000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00461\/110049\/ByT5-Towards-a-Token-Free-Future-with-Pre-trained"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":64,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00461","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}