{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T04:39:18Z","timestamp":1782535158910,"version":"3.54.5"},"reference-count":119,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2023,7,7]],"date-time":"2023-07-07T00:00:00Z","timestamp":1688688000000},"content-version":"vor","delay-in-days":187,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Byte-pair encoding (BPE) is widely used in NLP for performing subword tokenization. It uncovers redundant patterns for compressing the data, and hence alleviates the sparsity problem in downstream applications. Subwords discovered during the first merge operations tend to have the most substantial impact on the compression of texts. However, the structural underpinnings of this effect have not been analyzed cross-linguistically. We conduct in-depth analyses across 47 typologically diverse languages and three parallel corpora, and thereby show that the types of recurrent patterns that have the strongest impact on compression are an indicator of morphological typology. For languages with richer inflectional morphology there is a preference for highly productive subwords on the early merges, while for languages with less inflectional morphology, idiosyncratic subwords are more prominent. Both types of patterns contribute to efficient compression. Counter to the common perception that BPE subwords are not linguistically relevant, we find patterns across languages that resemble those described in traditional typology. We thus propose a novel way to characterize languages according to their BPE subword properties, inspired by the notion of morphological productivity in linguistics. This allows us to have language vectors that encode typological knowledge induced from raw text. Our approach is easily applicable to a wider range of languages and texts, as it does not require annotated data or any external linguistic knowledge. We discuss its potential contributions to quantitative typology and multilingual NLP.<\/jats:p>","DOI":"10.1162\/coli_a_00489","type":"journal-article","created":{"date-parts":[[2023,7,7]],"date-time":"2023-07-07T14:57:00Z","timestamp":1688741820000},"page":"943-1001","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":11,"title":["Languages Through the Looking Glass of BPE Compression"],"prefix":"10.1162","volume":"49","author":[{"given":"Ximena","family":"Gutierrez-Vasques","sequence":"first","affiliation":[{"name":"University of Z\u00fcrich, URPP Language and Space. ximena.gutierrezvasques@uzh.ch"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Bentz","sequence":"additional","affiliation":[{"name":"University of T\u00fcbingen, Department of General Linguistics. chris@christianbentz.de"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tanja","family":"Samard\u017ei\u0107","sequence":"additional","affiliation":[{"name":"University of Z\u00fcrich, URPP Language and Space, tanja.samardzic@uzh.ch"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,12,1]]},"reference":[{"key":"2024013020233252600_bib1","first-page":"1454","article-title":"Quechua","volume-title":"Morphology: An International Handbook of Inflection and Word-formation","author":"Adelaar","year":"2004"},{"key":"2024013020233252600_bib2","doi-asserted-by":"publisher","first-page":"3204","DOI":"10.18653\/v1\/P19-1310","article-title":"JW300: A wide-coverage parallel corpus for low-resource languages","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Agi\u0107","year":"2019"},{"key":"2024013020233252600_bib3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1017\/CBO9780511618437.001","article-title":"Typological distinctions in word-formation","volume-title":"Language Typology and Syntactic Description. Volume 3: Grammatical Categories and the Lexicon","author":"Aikhenvald","year":"2007"},{"key":"2024013020233252600_bib4","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1093\/oxfordhb\/9780199683208.013.18","article-title":"Polysynthetic structures of Lowland Amazonia","volume-title":"The Oxford Handbook of Polysynthesis","author":"Aikhenvald","year":"2017"},{"key":"2024013020233252600_bib5","unstructured":"Al-Rfou, Rami . 2015. Polyglot: A Massive Multilingual Natural Language Processing Pipeline. Ph.D. thesis, State University of New York at Stony Brook."},{"issue":"16","key":"2024013020233252600_bib6","doi-asserted-by":"publisher","first-page":"2627","DOI":"10.1016\/j.neuron.2021.06.009","article-title":"Mental compression of spatial sequences in human working memory using numerical and geometrical primitives","volume":"109","author":"Al Roumi","year":"2021","journal-title":"Neuron"},{"key":"2024013020233252600_bib7","doi-asserted-by":"publisher","first-page":"689","DOI":"10.18653\/v1\/2021.findings-emnlp.60","article-title":"How suitable are subword segmentation strategies for translating non-concatenative morphology?","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Amrhein","year":"2021"},{"key":"2024013020233252600_bib8","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1007\/978-94-011-2516-1_8","article-title":"Quantitative aspects of morphological productivity","volume-title":"Yearbook of Morphology 1991","author":"Baayen","year":"1992"},{"key":"2024013020233252600_bib9","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1007\/978-94-017-3710-4_7","article-title":"On frequency, transparency and productivity","volume-title":"Yearbook of Morphology 1992","author":"Baayen","year":"1993"},{"key":"2024013020233252600_bib10","first-page":"142","article-title":"A comparison between morphological complexity measures: Typological data vs. language corpora","volume-title":"Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (cl4lc)","author":"Bentz","year":"2016"},{"issue":"3","key":"2024013020233252600_bib11","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1080\/09296174.2014.911505","article-title":"On the relationship between type and token frequency","volume":"21","author":"Berg","year":"2014","journal-title":"Journal of Quantitative Linguistics"},{"key":"2024013020233252600_bib12","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1017\/CBO9780511618437.003","article-title":"Inflectional morphology","volume-title":"Language Typology and Syntactic Description. Volume 3: Grammatical Categories and the Lexicon","author":"Bickel","year":"2007"},{"key":"2024013020233252600_bib13","article-title":"Fusion of selected inflectional formatives","volume-title":"The World Atlas of Language Structures Online","author":"Bickel","year":"2013"},{"key":"2024013020233252600_bib14","unstructured":"Bickel, Balthasar, JohannaNichols, TarasZakharko, AlenaWitzlack-Makarevich, KristineHildebrandt, MichaelRie\u00dfler, LennartBierkandt, FernandoZ\u00fa\u00f1iga, and John B.Lowe. 2022. The autotyp database (v1.0.1)."},{"key":"2024013020233252600_bib15","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1093\/oxfordhb\/9780199683208.013.52","article-title":"The word in polysynthetic languages: Phonological and syntactic challenges","volume-title":"The Oxford Handbook of Polysynthesis","author":"Bickel","year":"2017"},{"key":"2024013020233252600_bib16","doi-asserted-by":"publisher","first-page":"907","DOI":"10.18653\/v1\/N18-1083","article-title":"From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Bjerva","year":"2018"},{"key":"2024013020233252600_bib17","doi-asserted-by":"publisher","first-page":"1529","DOI":"10.18653\/v1\/N19-1156","article-title":"A probabilistic generative model of linguistic typology","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Bjerva","year":"2019"},{"key":"2024013020233252600_bib18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2020.sigtyp-1.1","article-title":"SIGTYP 2020 shared task: Prediction of typological features","volume-title":"Proceedings of the Second Workshop on Computational Research in Linguistic Typology","author":"Bjerva","year":"2020"},{"key":"2024013020233252600_bib19","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1163\/9789004342934_008","article-title":"The Zipfian paradigm cell filling problem","volume-title":"Perspectives on Morphological Organization","author":"Blevins","year":"2017"},{"key":"2024013020233252600_bib20","doi-asserted-by":"publisher","first-page":"3563","DOI":"10.18653\/v1\/2022.emnlp-main.233","article-title":"Language contamination helps explains the cross-lingual capabilities of English pretrained models","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Blevins","year":"2022"},{"issue":"2","key":"2024013020233252600_bib21","doi-asserted-by":"publisher","first-page":"156","DOI":"10.3366\/word.2016.0092","article-title":"Joint predictiveness in inflectional paradigms","volume":"9","author":"Bonami","year":"2016","journal-title":"Word Structure"},{"key":"2024013020233252600_bib22","first-page":"15","article-title":"Sanuma","volume-title":"Handbook of Amazonian Languages 2","author":"Borgman","year":"1990"},{"key":"2024013020233252600_bib23","doi-asserted-by":"publisher","first-page":"4617","DOI":"10.18653\/v1\/2020.findings-emnlp.414","article-title":"Byte pair encoding is suboptimal for language model pretraining","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Bostrom","year":"2020"},{"key":"2024013020233252600_bib24","volume-title":"The Alamblak Language of Papua New Guinea (East Sepik)","author":"Bruce","year":"1984"},{"key":"2024013020233252600_bib25","volume-title":"Phonology and Language Use","author":"Bybee","year":"2003"},{"key":"2024013020233252600_bib26","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511750526","volume-title":"Language, Usage and Cognition","author":"Bybee","year":"2010"},{"issue":"3","key":"2024013020233252600_bib27","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1162\/coli_a_00444","article-title":"Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology","volume":"48","author":"Choenni","year":"2022","journal-title":"Computational Linguistics"},{"key":"2024013020233252600_bib28","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1162\/tacl_a_00448","article-title":"Canine: Pre-training an efficient tokenization-free encoder for language representation","volume":"10","author":"Clark","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024013020233252600_bib29","doi-asserted-by":"publisher","first-page":"21","DOI":"10.3115\/1118647.1118650","article-title":"Unsupervised discovery of morphemes","volume-title":"Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning","author":"Creutz","year":"2002"},{"key":"2024013020233252600_bib30","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1016\/j.jml.2015.06.005","article-title":"The adoption of linguistic rules in native and non-native speakers: Evidence from a Wug task","volume":"84","author":"Cuskley","year":"2015","journal-title":"Journal of Memory and Language"},{"issue":"2","key":"2024013020233252600_bib31","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1524\/stuf.2007.60.2.95","article-title":"Parallel texts: Using translational equivalents in linguistic typology","volume":"60","author":"Cysouw","year":"2007","journal-title":"STUF-Sprachtypologie und Universalienforschung"},{"key":"2024013020233252600_bib32","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1093\/oxfordhb\/9780199683208.013.3","article-title":"Polysynthesis and complexity","volume-title":"The Oxford Handbook of Polysynthesis","author":"Dahl","year":"2017"},{"key":"2024013020233252600_bib33","volume-title":"The World\u2019s Writing Systems","author":"Daniels","year":"1996"},{"key":"2024013020233252600_bib34","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2024013020233252600_bib35","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511486241","volume-title":"Word: A cross-linguistic Typology","author":"Dixon","year":"2003"},{"key":"2024013020233252600_bib36","volume-title":"A Grammar of Boumaa Fijian","author":"Dixon","year":"1988"},{"key":"2024013020233252600_bib37","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1007\/978-3-031-24337-0_38","article-title":"How much does tokenization affect neural machine translation?","volume-title":"Computational Linguistics and Intelligent Text Processing","author":"Domingo","year":"2023"},{"key":"2024013020233252600_bib38","article-title":"Prefixing vs. suffixing in inflectional morphology","volume-title":"The World Atlas of Language Structures Online","author":"Dryer","year":"2013"},{"key":"2024013020233252600_bib39","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7385533","volume-title":"The World Atlas of Language Structures Online","author":"Dryer","year":"2013"},{"key":"2024013020233252600_bib40","doi-asserted-by":"crossref","unstructured":"Ehret, Katharina . 2016. An Information-Theoretic Approach to Language Complexity: Variation in Naturalistic Corpora. Ph.D. thesis, Albert-Ludwigs-Universit\u00e4t Freiburg.","DOI":"10.1515\/9783110348965-004"},{"key":"2024013020233252600_bib41","doi-asserted-by":"publisher","DOI":"10.1515\/9783110348965-004","article-title":"An information-theoretic approach to assess linguistic complexity","volume-title":"Complexity, Isolation and Variation","author":"Ehret","year":"2016"},{"key":"2024013020233252600_bib42","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1163\/9789004257023_007","article-title":"Negation in Apurin\u00e3 (Arawak)","volume-title":"Negation in Arawak languages","author":"Facundes","year":"2014"},{"issue":"3","key":"2024013020233252600_bib43","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1080\/09296174.2017.1366095","article-title":"Optimization models of natural communication","volume":"25","author":"Cancho","year":"2018","journal-title":"Journal of Quantitative Linguistics"},{"issue":"2","key":"2024013020233252600_bib44","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1080\/09296174.2020.1778387","article-title":"Optimal coding and the origins of Zipfian laws","volume":"29","author":"Cancho","year":"2022","journal-title":"Journal of Quantitative Linguistics"},{"issue":"8","key":"2024013020233252600_bib45","doi-asserted-by":"publisher","first-page":"1565","DOI":"10.1111\/cogs.12061","article-title":"Compression as a universal principle of animal behavior","volume":"37","author":"Cancho","year":"2013","journal-title":"Cognitive Science"},{"issue":"1","key":"2024013020233252600_bib46","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1080\/03740463.1992.10412275","article-title":"The development of morphophonemic complexity in eskimo languages","volume":"25","author":"Fortescue","year":"1992","journal-title":"Acta Linguistica Hafniensia"},{"key":"2024013020233252600_bib47","volume-title":"West Greenlandic","author":"Fortescue","year":"1984"},{"issue":"2","key":"2024013020233252600_bib48","first-page":"23","article-title":"A new algorithm for data compression","volume":"12","author":"Gage","year":"1994","journal-title":"The C Users Journal"},{"key":"2024013020233252600_bib49","doi-asserted-by":"publisher","first-page":"1375","DOI":"10.18653\/v1\/D19-1141","article-title":"Investigating the effectiveness of BPE: The power of shorter sequences","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Gall\u00e9","year":"2019"},{"issue":"1","key":"2024013020233252600_bib50","first-page":"25","article-title":"The informativeness of linguistic unit boundaries","author":"Geertzen","year":"2016","journal-title":"Italian Journal of Linguistics"},{"key":"2024013020233252600_bib51","doi-asserted-by":"publisher","DOI":"10.4324\/9780203340769","volume-title":"Turkish: A Comprehensive Grammar","author":"G\u00f6ksel","year":"2004"},{"issue":"2","key":"2024013020233252600_bib52","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1162\/089120101750300490","article-title":"Unsupervised learning of the morphology of a natural language","volume":"27","author":"Goldsmith","year":"2001","journal-title":"Computational Linguistics"},{"issue":"30","key":"2024013020233252600_bib53","first-page":"43","article-title":"Fonolog\u00eda de dos idiomas tukano del piraparan\u00e1: Barasana y tatuyo","volume":"29","author":"Gomez-Imbert","year":"2004","journal-title":"Amerindia"},{"key":"2024013020233252600_bib54","article-title":"Language Universals: With special reference to feature hierarchies","author":"Greenberg","year":"1966"},{"issue":"3","key":"2024013020233252600_bib55","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1086\/464575","article-title":"A quantitative approach to the morphological typology of language","volume":"26","author":"Greenberg","year":"1960","journal-title":"International Journal of American Linguistics"},{"key":"2024013020233252600_bib56","first-page":"1177","article-title":"Morfessor FlatCat: An HMM-based method for unsupervised and semi-supervised learning of morphology","volume-title":"Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers","author":"Gr\u00f6nroos","year":"2014"},{"issue":"2","key":"2024013020233252600_bib57","doi-asserted-by":"crossref","first-page":"119","DOI":"10.62190\/amla.lmne.2019.1.2.7","article-title":"Supleci\u00f3n en yaqui y wix\u00e1rika","volume":"1","author":"Guerrero","year":"2019","journal-title":"Ling\u00fc\u00edstica Mexicana"},{"key":"2024013020233252600_bib58","doi-asserted-by":"publisher","first-page":"3454","DOI":"10.18653\/v1\/2021.eacl-main.302","article-title":"From characters to words: The turning point of BPE merges","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Gutierrez-Vasques","year":"2021"},{"key":"2024013020233252600_bib59","doi-asserted-by":"publisher","DOI":"10.1515\/9783110194036","volume-title":"Language Typology and Language Universals \/ Sprachtypologie und sprachliche Universalien \/ La typologie des langues et les universaux linguistiques: Eu - L","author":"Haspelmath","year":"2008"},{"issue":"s1000","key":"2024013020233252600_bib60","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1515\/flin-2017-1005","article-title":"The indeterminacy of word segmentation and the nature of morphology and syntax","volume":"51","author":"Haspelmath","year":"2017","journal-title":"Folia Linguistica"},{"key":"2024013020233252600_bib61","volume-title":"Understanding Morphology","author":"Haspelmath","year":"2010"},{"key":"2024013020233252600_bib62","doi-asserted-by":"publisher","DOI":"10.1075\/loall.2","volume-title":"Georgian: A Structural Reference Grammar","author":"Hewitt","year":"1995"},{"key":"2024013020233252600_bib63","doi-asserted-by":"publisher","DOI":"10.4324\/9781315651194","volume-title":"Burmese: A Comprehensive Grammar","author":"Jenny","year":"2016"},{"issue":"11","key":"2024013020233252600_bib64","doi-asserted-by":"publisher","first-page":"e2113883119","DOI":"10.1073\/pnas.2113883119","article-title":"Symmetry and simplicity spontaneously emerge from the algorithmic nature of evolution","volume":"119","author":"Johnston","year":"2022","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"3","key":"2024013020233252600_bib65","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1080\/09296179808590128","article-title":"Measuring linguistic complexity: The morphological tier","volume":"5","author":"Juola","year":"1998","journal-title":"Journal of Quantitative Linguistics"},{"key":"2024013020233252600_bib66","unstructured":"Karan, Elke . 2006. Writing system development and reform: A process. MA thesis, University of North Dakota. Available athttps:\/\/commons.und.edu\/theses\/3004"},{"key":"2024013020233252600_bib67","first-page":"1","article-title":"The type-token relationship in slavic parallel texts","volume":"20","author":"Kelih","year":"2010","journal-title":"Glottometrics"},{"key":"2024013020233252600_bib68","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.cognition.2015.03.016","article-title":"Compression and communication in the cultural evolution of linguistic structure","volume":"141","author":"Kirby","year":"2015","journal-title":"Cognition"},{"key":"2024013020233252600_bib69","doi-asserted-by":"publisher","first-page":"4","DOI":"10.3389\/neuro.06.004.2008","article-title":"Representational similarity analysis\u2014Connecting the branches of systems neuroscience","author":"Kriegeskorte","year":"2008","journal-title":"Frontiers in Systems Neuroscienc"},{"key":"2024013020233252600_bib70","doi-asserted-by":"publisher","first-page":"66","DOI":"10.18653\/v1\/P18-1007","article-title":"Subword regularization: Improving neural network translation models with multiple subword candidates","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Kudo","year":"2018"},{"key":"2024013020233252600_bib71","doi-asserted-by":"publisher","first-page":"4483","DOI":"10.18653\/v1\/2020.emnlp-main.363","article-title":"From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Lauscher","year":"2020"},{"key":"2024013020233252600_bib72","doi-asserted-by":"publisher","first-page":"3125","DOI":"10.18653\/v1\/P19-1301","article-title":"Choosing transfer languages for cross-lingual learning","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Lin","year":"2019"},{"key":"2024013020233252600_bib73","doi-asserted-by":"publisher","first-page":"8","DOI":"10.18653\/v1\/E17-2002","volume-title":"URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors","author":"Littell","year":"2017"},{"key":"2024013020233252600_bib74","volume-title":"Burmese Grammar and Grammatical Analysis","author":"Lonsdale","year":"1899"},{"key":"2024013020233252600_bib75","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1007\/978-3-030-00794-2_30","article-title":"Morphological and language-agnostic word segmentation for NMT","volume-title":"International Conference on Text, Speech, and Dialogue","author":"Mach\u00e1\u010dek","year":"2018"},{"key":"2024013020233252600_bib76","doi-asserted-by":"publisher","first-page":"961","DOI":"10.18653\/v1\/2022.findings-acl.78","article-title":"BPE vs. morphological segmentation: A case study on machine translation of four polysynthetic languages","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Mager","year":"2022"},{"key":"2024013020233252600_bib77","doi-asserted-by":"publisher","first-page":"2529","DOI":"10.18653\/v1\/D17-1268","article-title":"Learning language representations for typology prediction","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Malaviya","year":"2017"},{"key":"2024013020233252600_bib78","doi-asserted-by":"publisher","first-page":"4903","DOI":"10.18653\/v1\/2022.naacl-main.361","article-title":"A balanced data approach for evaluating cross-lingual transfer: Mapping the linguistic blood bank","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Malkin","year":"2022"},{"issue":"273","key":"2024013020233252600_bib79","first-page":"40","article-title":"Creating a massively parallel bible corpus","volume":"135","author":"Mayer","year":"2014","journal-title":"Oceania"},{"key":"2024013020233252600_bib80","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1075\/slcs.150.02may","article-title":"From the extraction of continuous features in parallel texts to visual analytics of heterogeneous areal-typological datasets","volume-title":"The Role of Functionally Oriented Computational Models","author":"Mayer","year":"2014"},{"key":"2024013020233252600_bib81","article-title":"Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP","author":"Mielke","year":"2021","journal-title":"arXiv preprint arXiv:2112.10508"},{"key":"2024013020233252600_bib82","doi-asserted-by":"publisher","first-page":"6843","DOI":"10.1609\/aaai.v33i01.33016843","article-title":"Spell once, summon anywhere: A two-level open-vocabulary language model","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Mielke","year":"2019"},{"key":"2024013020233252600_bib83","volume-title":"The Unicode Cookbook for Linguists: Managing Writing Systems Using Orthography Profiles","author":"Moran","year":"2018"},{"key":"2024013020233252600_bib84","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511978876","volume-title":"Introducing Language Typology","author":"Moravcsik","year":"2012"},{"key":"2024013020233252600_bib85","doi-asserted-by":"publisher","first-page":"2453","DOI":"10.1016\/B0-08-043076-7\/00589-1","article-title":"Computational approaches to model evaluation","volume-title":"International Encyclopedia of the Social & Behavioral Sciences","author":"Myung","year":"2001"},{"key":"2024013020233252600_bib86","doi-asserted-by":"publisher","first-page":"5347","DOI":"10.18653\/v1\/2022.acl-long.367","article-title":"KinyaBERT: A morphology-aware Kinyarwanda language model","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Nzeyimana","year":"2022"},{"key":"2024013020233252600_bib87","doi-asserted-by":"publisher","first-page":"1308","DOI":"10.18653\/v1\/2022.naacl-main.94","article-title":"Quantifying synthesis and fusion and their impact on machine translation","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Oncevay","year":"2022"},{"key":"2024013020233252600_bib88","doi-asserted-by":"publisher","first-page":"205","DOI":"10.3115\/v1\/P15-2034","article-title":"Word order typology through multilingual word alignment","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"\u00d6stling","year":"2015"},{"key":"2024013020233252600_bib89","doi-asserted-by":"publisher","DOI":"10.1515\/9783110295252","volume-title":"The Languages and Linguistics of the New Guinea Area: A Comprehensive Guide","author":"Palmer","year":"2017"},{"key":"2024013020233252600_bib90","first-page":"727","article-title":"X-means: Extending k-means with efficient estimation of the number of clusters","volume-title":"ICML","author":"Pelleg","year":"2000"},{"key":"2024013020233252600_bib91","doi-asserted-by":"publisher","first-page":"7428","DOI":"10.18653\/v1\/2022.emnlp-main.503","article-title":"Subword evenness (sue) as a predictor of cross-lingual transfer to low-resource languages","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Main Volume","author":"Pelloni","year":"2022"},{"issue":"5019","key":"2024013020233252600_bib92","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1126\/science.1857983","article-title":"Rules of language","volume":"253","author":"Pinker","year":"1991","journal-title":"Science"},{"key":"2024013020233252600_bib93","doi-asserted-by":"publisher","first-page":"4996","DOI":"10.18653\/v1\/P19-1493","article-title":"How multilingual is multilingual BERT?","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Pires","year":"2019"},{"issue":"3","key":"2024013020233252600_bib94","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1162\/coli_a_00357","article-title":"Modeling language variation and universals: A survey on typological linguistics for natural language processing","volume":"45","author":"Ponti","year":"2019","journal-title":"Computational Linguistics"},{"issue":"8","key":"2024013020233252600_bib95","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2024013020233252600_bib96","doi-asserted-by":"publisher","first-page":"10115","DOI":"10.18653\/v1\/2021.emnlp-main.793","article-title":"An information-theoretic characterization of morphological fusion","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Rathi","year":"2021"},{"key":"2024013020233252600_bib97","article-title":"Lempel-Ziv data compression technique utilizing a dictionary pre-filled with frequent letter combinations, words and\/or phrases","author":"Reynar","year":"1999"},{"issue":"5","key":"2024013020233252600_bib98","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1016\/0005-1098(78)90005-5","article-title":"Modeling by shortest data description","volume":"14","author":"Rissanen","year":"1978","journal-title":"Automatica"},{"key":"2024013020233252600_bib99","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"Rousseeuw","year":"1987","journal-title":"Journal of Computational and Applied Mathematics"},{"key":"2024013020233252600_bib100","doi-asserted-by":"publisher","first-page":"10215","DOI":"10.18653\/v1\/2021.emnlp-main.802","article-title":"XTREME-R: Towards more challenging and nuanced multilingual evaluation","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Ruder","year":"2021"},{"key":"2024013020233252600_bib101","doi-asserted-by":"publisher","first-page":"3118","DOI":"10.18653\/v1\/2021.acl-long.243","article-title":"How good is your tokenizer? On the monolingual performance of multilingual language models","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Rust","year":"2021"},{"key":"2024013020233252600_bib102","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511486975","volume-title":"A Reference Grammar of Modern Standard Arabic","author":"Ryding","year":"2005"},{"key":"2024013020233252600_bib103","doi-asserted-by":"publisher","first-page":"164","DOI":"10.18653\/v1\/2021.eacl-srw.22","article-title":"The effectiveness of morphology-aware segmentation in low-resource neural machine translation","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop","author":"Saleva","year":"2021"},{"key":"2024013020233252600_bib104","volume-title":"Language: An Introduction to the Study of Speech","author":"Sapir","year":"1921"},{"key":"2024013020233252600_bib105","doi-asserted-by":"publisher","first-page":"5149","DOI":"10.1109\/ICASSP.2012.6289079","article-title":"Japanese and Korean voice search","volume-title":"2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","author":"Schuster","year":"2012"},{"key":"2024013020233252600_bib106","doi-asserted-by":"crossref","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with subword units","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2024013020233252600_bib107","article-title":"BPE and charCNNs for translation of morphology: A cross-lingual comparison and analysis","author":"Shapiro","year":"2018","journal-title":"arXiv preprint arXiv:1809.01301"},{"key":"2024013020233252600_bib108","volume-title":"A Grammar of Yauyos Quechua","author":"Shimelman","year":"2017"},{"key":"2024013020233252600_bib109","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordhb\/9780199591428.013.20","article-title":"Chamorro inflection","volume-title":"The Oxford Handbook of Inflection","author":"Stolz","year":"2015"},{"key":"2024013020233252600_bib110","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1145\/800133.804329","article-title":"The macro model for data compression","volume-title":"Proceedings of the Tenth Annual ACM Symposium on Theory of Computing","author":"Storer","year":"1978"},{"key":"2024013020233252600_bib111","volume-title":"Inflection","author":"Stump","year":"2017"},{"issue":"1","key":"2024013020233252600_bib112","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1111\/cogs.12144","article-title":"Culture: Copying, compression, and conventionality","volume":"39","author":"Tamariz","year":"2015","journal-title":"Cognitive Science"},{"issue":"1","key":"2024013020233252600_bib113","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1080\/016909699386374","article-title":"Acceptability ratings of regular and irregular past-tense forms: Evidence for a dual-system model of language from word frequency and phonological neighbourhood effects","volume":"14","author":"Ullman","year":"1999","journal-title":"Language and Cognitive Processes"},{"key":"2024013020233252600_bib114","doi-asserted-by":"publisher","first-page":"2302","DOI":"10.18653\/v1\/2020.emnlp-main.180","article-title":"UDapter: Language adaptation for truly Universal Dependency parsing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"\u00dcst\u00fcn","year":"2020"},{"issue":"3","key":"2024013020233252600_bib115","doi-asserted-by":"publisher","first-page":"671","DOI":"10.1515\/ling-2012-0021","article-title":"Lexical typology through similarity semantics: Toward a semantic map of motion verbs","volume":"50","author":"W\u00e4lchli","year":"2012","journal-title":"Linguistics"},{"key":"2024013020233252600_bib116","doi-asserted-by":"publisher","first-page":"5117","DOI":"10.18653\/v1\/P19-1505","article-title":"Morphological irregularity correlates with frequency","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Wu","year":"2019"},{"key":"2024013020233252600_bib117","volume-title":"Korean: A Comprehensive Grammar","author":"Yeon","year":"2011"},{"issue":"3","key":"2024013020233252600_bib118","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1109\/TIT.1977.1055714","article-title":"A universal algorithm for sequential data compression","volume":"23","author":"Ziv","year":"1977","journal-title":"IEEE Transactions on Information Theory"},{"key":"2024013020233252600_bib119","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1093\/oxfordhb\/9780199683208.013.40","article-title":"Mapudungun","volume-title":"The Oxford Handbook of Polysynthesis","author":"Z\u00fa\u00f1iga","year":"2017"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/49\/4\/943\/2269507\/coli_a_00489.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/49\/4\/943\/2269507\/coli_a_00489.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,23]],"date-time":"2024-10-23T20:29:59Z","timestamp":1729715399000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/49\/4\/943\/116635\/Languages-Through-the-Looking-Glass-of-BPE"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":119,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,12,1]]},"published-print":{"date-parts":[[2023,12,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00489","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}