{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,24]],"date-time":"2026-08-24T15:05:40Z","timestamp":1787583940473,"version":"build-2736575974"},"reference-count":60,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T00:00:00Z","timestamp":1684454400000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,5,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Tokenization is fundamental to pretrained language models (PLMs). Existing tokenization methods for Chinese PLMs typically treat each character as an indivisible token. However, they ignore the unique feature of the Chinese writing system where additional linguistic information exists below the character level, i.e., at the sub-character level. To utilize such information, we propose sub-character (SubChar for short) tokenization. Specifically, we first encode the input text by converting each Chinese character into a short sequence based on its glyph or pronunciation, and then construct the vocabulary based on the encoded text with sub-word segmentation. Experimental results show that SubChar tokenizers have two main advantages over existing tokenizers: 1) They can tokenize inputs into much shorter sequences, thus improving the computational efficiency. 2) Pronunciation-based SubChar tokenizers can encode Chinese homophones into the same transliteration sequences and produce the same tokenization output, hence being robust to homophone typos. At the same time, models trained with SubChar tokenizers perform competitively on downstream tasks. We release our code and models at https:\/\/github.com\/thunlp\/SubCharTokenization to facilitate future work.<\/jats:p>","DOI":"10.1162\/tacl_a_00560","type":"journal-article","created":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T19:28:57Z","timestamp":1684524537000},"page":"469-487","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":13,"title":["Sub-Character Tokenization for Chinese Pretrained Language\n                    Models"],"prefix":"10.1162","volume":"11","author":[{"given":"Chenglei","family":"Si","sequence":"first","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. clsi@terpmail.umd.edu"},{"name":"University of Maryland, College Park, MD, USA. clsi@terpmail.umd.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengyan","family":"Zhang","sequence":"additional","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. zy-z19@mails.tsinghua.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingfa","family":"Chen","sequence":"additional","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. yingfa-c18@mails.tsinghua.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fanchao","family":"Qi","sequence":"additional","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. qfc17@mails.tsinghua.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaozhi","family":"Wang","sequence":"additional","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. wangxz20@mails.tsinghua.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiyuan","family":"Liu","sequence":"additional","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. liuzy@tsinghua.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Huawei Noah\u2019s Ark Lab, Hong Kong, China. wangyasheng@huawei.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qun","family":"Liu","sequence":"additional","affiliation":[{"name":"Huawei Noah\u2019s Ark Lab, Hong Kong, China. qun.liu@huawei.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maosong","family":"Sun","sequence":"additional","affiliation":[{"name":"NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. sms@tsinghua.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,5,18]]},"reference":[{"key":"2023051919285165600_bib1","article-title":"The #BenderRule: On naming the languages\n                        we study and why it matters","author":"Bender","year":"2019","journal-title":"The\n                    Gradient"},{"key":"2023051919285165600_bib2","doi-asserted-by":"publisher","first-page":"1462","DOI":"10.18653\/v1\/D16-1153","article-title":"Phonologically aware neural model for\n                        named entity recognition in low resource transfer settings","volume-title":"Proceedings of EMNLP","author":"Bharadwaj","year":"2016"},{"key":"2023051919285165600_bib3","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.12029","article-title":"cw2vec: Learning chinese word embeddings with stroke n-gram\n                        information","volume-title":"Proceedings of\n                    AAAI","author":"Cao","year":"2018"},{"key":"2023051919285165600_bib4","doi-asserted-by":"publisher","DOI":"10.3115\/1626394.1626430","article-title":"Optimizing Chinese word segmentation for\n                        machine translation performance","volume-title":"Proceedings of\n                        the Third Workshop on Statistical Machine Translation","author":"Chang","year":"2008"},{"key":"2023051919285165600_bib5","doi-asserted-by":"publisher","first-page":"3285","DOI":"10.18653\/v1\/D18-1366","article-title":"Adapting word embeddings to new languages\n                        with morphological and phonological subword representations","volume-title":"Proceedings of EMNLP","author":"Chaudhary","year":"2018"},{"key":"2023051919285165600_bib6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1536","article-title":"The BQ corpus: A large-scale domain-specific Chinese corpus\n                        for sentence semantic equivalence identification","volume-title":"Proceedings of EMNLP","author":"Chen","year":"2018"},{"key":"2023051919285165600_bib7","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1162\/tacl_a_00448","article-title":"CANINE: Pre-training an efficient\n                        tokenization-free encoder for language representation","volume":"10","author":"Clark","year":"2021","journal-title":"Transactions of the Association for Computational\n                        Linguistics"},{"key":"2023051919285165600_bib8","article-title":"ELECTRA: Pre-training text encoders as\n                        discriminators rather than generators","volume-title":"Proceedings of ICLR","author":"Clark","year":"2020"},{"key":"2023051919285165600_bib9","volume-title":"The Writing Systems of the World","author":"Coulmas","year":"1991"},{"key":"2023051919285165600_bib10","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.findings-emnlp.58","article-title":"Revisiting pre-trained models for Chinese\n                        natural language processing","volume-title":"Findings of\n                        EMNLP","author":"Cui","year":"2020"},{"key":"2023051919285165600_bib11","first-page":"3504","article-title":"Pre-training with whole word masking for\n                        Chinese BERT","volume":"29","author":"Cui","year":"2019","journal-title":"IEEE\/ACM TASLP"},{"key":"2023051919285165600_bib12","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D19-1600","article-title":"A span-extraction dataset for Chinese\n                        machine reading comprehension","volume-title":"Proceedings of\n                        EMNLP-IJCNLP","author":"Cui","year":"2019"},{"key":"2023051919285165600_bib13","article-title":"BERT: Pre-training of deep bidirectional\n                        transformers for language understanding","volume-title":"Proceedings of NAACL-HLT","author":"Devlin","year":"2019"},{"key":"2023051919285165600_bib14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.425","article-title":"ZEN: Pre-training Chinese text encoder enhanced by n-gram\n                        representations","volume-title":"Findings of EMNLP","author":"Diao","year":"2020"},{"key":"2023051919285165600_bib15","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-50496-4_20","article-title":"Character-based lstm-crf with radical-level features for\n                        chinese named entity recognition","volume-title":"International\n                        Conference on Computer Processing of Oriental Languages","author":"Dong","year":"2016"},{"key":"2023051919285165600_bib16","doi-asserted-by":"crossref","DOI":"10.1017\/S0142716421000114","article-title":"The effect of second-language orthographic input on the\n                        phonological encoding of Mandarin words","author":"Hao","year":"2021","journal-title":"Applied\n                        Psycholinguistics"},{"key":"2023051919285165600_bib17","article-title":"DeBERTa: Decoding-enhanced BERT with disentangled\n                        attention","volume-title":"Proceedings of\n                    ICLR","author":"He","year":"2021"},{"key":"2023051919285165600_bib18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.314","article-title":"OCNLI: Original Chinese natural language\n                        inference","volume-title":"Findings of EMNLP","author":"Hai","year":"2020"},{"key":"2023051919285165600_bib19","doi-asserted-by":"publisher","first-page":"5958","DOI":"10.18653\/v1\/2021.acl-long.464","article-title":"PHMOSpell: Phonological and morphological knowledge guided\n                        Chinese spelling check","volume-title":"Proceedings of\n                        ACL","author":"Li","year":"2021"},{"key":"2023051919285165600_bib20","article-title":"Efficient sequence packing without\n                        cross-contamination: Accelerating large language models without impacting\n                        performance","volume":"abs\/2107.02027","author":"Krell","year":"2021","journal-title":"arXiv preprint"},{"key":"2023051919285165600_bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1007","article-title":"Subword regularization: Improving neural network translation\n                        models with multiple subword candidates","volume-title":"Proceedings of ACL","author":"Kudo","year":"2018"},{"key":"2023051919285165600_bib22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-2012","article-title":"SentencePiece: A simple and language\n                        independent subword tokenizer and detokenizer for neural text\n                        processing","volume-title":"Proceedings of EMNLP System\n                        Demonstrations","author":"Kudo","year":"2018"},{"key":"2023051919285165600_bib23","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2021.naacl-main.137","article-title":"Lattice-BERT: Leveraging multi-granularity representations in\n                        Chinese pre-trained language models","volume-title":"Proceedings\n                        of NAACL-HLT","author":"Lai","year":"2021"},{"key":"2023051919285165600_bib24","article-title":"ALBERT: A lite BERT for self-supervised\n                        learning of language representations","volume-title":"Proceedings\n                        of ICLR","author":"Lan","year":"2020"},{"key":"2023051919285165600_bib25","article-title":"The Winograd schema\n                        challenge","volume-title":"Thirteenth International Conference on\n                        the Principles of Knowledge Representation and\n                    Reasoning","author":"Levesque","year":"2012"},{"key":"2023051919285165600_bib26","article-title":"Scalable term selection for text\n                        categorization","volume-title":"Proceedings of\n                    EMNLP","author":"Li","year":"2007"},{"key":"2023051919285165600_bib27","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1314","article-title":"Is word segmentation necessary for deep learning of Chinese\n                        representations?","volume-title":"Proceedings of\n                    ACL","author":"Li","year":"2019"},{"key":"2023051919285165600_bib28","doi-asserted-by":"crossref","DOI":"10.1162\/coli.2009.35.4.35403","article-title":"Punctuation as implicit annotations for Chinese word\n                        segmentation","author":"Li","year":"2009","journal-title":"Computational\n                    Linguistics"},{"key":"2023051919285165600_bib29","first-page":"2059","article-title":"Learning character-level compositionality\n                        with visual features","volume-title":"Proceedings of\n                    ACL","author":"Liu","year":"2017"},{"key":"2023051919285165600_bib30","article-title":"RoBERTa: A robustly optimized BERT\n                        pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv preprint"},{"key":"2023051919285165600_bib31","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.coling-main.4","article-title":"CharBERT: Character-aware pre-trained\n                        language model","volume-title":"Proceedings of\n                    COLING","author":"Ma","year":"2020"},{"key":"2023051919285165600_bib32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1334","article-title":"Right for the wrong reasons: Diagnosing\n                        syntactic heuristics in natural language inference","volume-title":"Proceedings of ACL","author":"Thomas McCoy","year":"2019"},{"key":"2023051919285165600_bib33","article-title":"Glyce: Glyph-vectors for Chinese character\n                        representations","volume-title":"Proceedings of\n                    NeurIPS","author":"Meng","year":"2019"},{"key":"2023051919285165600_bib34","article-title":"Distributed representations of words and phrases and their\n                        compositionality","volume-title":"Proceedings of\n                    NeurIPS","author":"Mikolov","year":"2013"},{"key":"2023051919285165600_bib35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.367","article-title":"KinyaBERT: A morphology-aware kinyarwanda\n                        language model","volume-title":"Proceedings of ACL","author":"Nzeyimana","year":"2022"},{"key":"2023051919285165600_bib36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word\n                        representations","volume-title":"Proceedings of\n                        NAACL-HLT","author":"Peters","year":"2018"},{"key":"2023051919285165600_bib37","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2012.6289079","article-title":"Japanese and Korean voice\n                        search","volume-title":"Proceedings of the IEEE International\n                        Conference on Acoustics, Speech and Signal Processing","author":"Schuster","year":"2012"},{"key":"2023051919285165600_bib38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with\n                        subword units","volume-title":"Proceedings of ACL","author":"Sennrich","year":"2016"},{"key":"2023051919285165600_bib39","article-title":"READIN: A Chinese multi-task benchmark with realistic and\n                        diverse input noises","author":"Si","year":"2023","journal-title":"arXiv"},{"key":"2023051919285165600_bib40","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00305","article-title":"Investigating prior knowledge for\n                        challenging Chinese machine reading comprehension","author":"Sun","year":"2020","journal-title":"Transactions of the Association for Computational\n                        Linguistics"},{"key":"2023051919285165600_bib41","article-title":"THULAC: An efficient lexical analyzer for\n                        Chinese","author":"Sun","year":"2016","journal-title":"GitHub"},{"key":"2023051919285165600_bib42","first-page":"279","article-title":"Radical-enhanced chinese character embedding","volume-title":"Proceedings of COLING","author":"Sun","year":"2014"},{"key":"2023051919285165600_bib43","article-title":"ERNIE: Enhanced representation through\n                        knowledge integration","author":"Sun","year":"2019","journal-title":"arXiv preprint"},{"key":"2023051919285165600_bib44","doi-asserted-by":"publisher","first-page":"2065","DOI":"10.18653\/v1\/2021.acl-long.161","article-title":"ChineseBERT: Chinese pretraining enhanced by glyph and Pinyin\n                        information","volume-title":"Proceedings of ACL","author":"Sun","year":"2021"},{"key":"2023051919285165600_bib45","article-title":"Charformer: Fast character transformers\n                        via gradient-based subword tokenization","volume-title":"Proceedings of ICLR","author":"Yi","year":"2022"},{"key":"2023051919285165600_bib46","article-title":"Attention is all you need","volume-title":"Proceedings of NeurIPS","author":"Vaswani","year":"2017"},{"key":"2023051919285165600_bib47","article-title":"Training multilingual pre-trained language\n                        model with byte-level \u2223 subwords","author":"Wei","year":"2021","journal-title":"arXiv\n                        preprint"},{"key":"2023051919285165600_bib48","article-title":"NEZHA: Neural contextualized representation for Chinese\n                        language understanding","author":"Wei","year":"2019","journal-title":"arXiv"},{"key":"2023051919285165600_bib49","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.136","article-title":"ERNIE-Gram: Pre-training with explicitly n-gram masked\n                        language modeling for natural language understanding","volume-title":"Proceedings of NAACL-HLT","author":"Xiao","year":"2021"},{"key":"2023051919285165600_bib50","article-title":"CLUENER2020: Fine-grained name entity\n                        recognition for Chinese","author":"Liang","year":"2020","journal-title":"arXiv preprint"},{"key":"2023051919285165600_bib51","article-title":"CLUE: A Chinese language understanding evaluation\n                        benchmark","volume-title":"Proceedings of\n                    COLING","author":"Liang","year":"2020"},{"key":"2023051919285165600_bib52","doi-asserted-by":"crossref","DOI":"10.1162\/tacl_a_00461","article-title":"ByT5: Towards a token-free future with\n                        pre-trained byte-to-byte models","author":"Xue","year":"2022","journal-title":"Transactions of the\n                        Association for Computational Linguistics"},{"key":"2023051919285165600_bib53","first-page":"286","article-title":"Joint embeddings of Chinese words, characters, and\n                        fine-grained subcharacter components","volume-title":"Proceedings\n                        of EMNLP","author":"Jinxing","year":"2017"},{"key":"2023051919285165600_bib54","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.37","article-title":"AMBERT: A pre-trained language model with multi-grained\n                        tokenization","volume-title":"Findings of ACL","author":"Zhang","year":"2021"},{"key":"2023051919285165600_bib55","article-title":"PAWS: Paraphrase adversaries from word\n                        scrambling","volume-title":"Proceedings of\n                    NAACL-HLT","author":"Zhang","year":"2019"},{"key":"2023051919285165600_bib56","doi-asserted-by":"publisher","first-page":"1011","DOI":"10.1145\/3357384.3358005","article-title":"Learning chinese word embeddings from stroke,\n                        structure and pinyin of characters","volume-title":"Proceedings\n                        of CIKM","author":"Zhang","year":"2019"},{"key":"2023051919285165600_bib57","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.12.003","article-title":"CPM-2: Large-scale cost-effective pre-trained language\n                        models","author":"Zhang","year":"2021","journal-title":"arXiv preprint"},{"key":"2023051919285165600_bib58","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.07.001","article-title":"CPM: A large-scale generative Chinese pre-trained language\n                        model","author":"Zhang","year":"2020","journal-title":"AI Open"},{"key":"2023051919285165600_bib59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1075","article-title":"ChID: A large-scale Chinese IDiom dataset for cloze\n                        test","volume-title":"Proceedings of ACL","author":"Zheng","year":"2019"},{"key":"2023051919285165600_bib60","article-title":"MVP-BERT: Redesigning vocabularies for Chinese BERT and\n                        multi-vocab pretraining","author":"Zhu","year":"2020","journal-title":"arXiv preprint"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00560\/2102899\/tacl_a_00560.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00560\/2102899\/tacl_a_00560.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T19:29:17Z","timestamp":1684524557000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00560\/116047\/Sub-Character-Tokenization-for-Chinese-Pretrained"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,18]]},"references-count":60,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00560","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,18]]}}}